Kimi K3 vLLM Error Reproduction: Build a Stable Baseline

A single parameter change turns a CUDA error into an OOM, then a cache warning, then an NCCL failure.

Fastest fix: stop changing the driver, image, and inference parameters together. Freeze the failed environment first, then reproduce in this order: compatibility, minimal startup, one request, and one optional feature at a time.

Who should use this guide

This is for infrastructure engineers who have already restarted Kimi K3 several times and now have no stable failure signature. It also fits Agent platform teams collecting a reproducible case for an upstream maintainer, and technical leads deciding whether to keep consuming a shared cluster or move the test to isolated capacity.

Last updated: August 9, 2026. Compatibility data was checked against the official Kimi K3 recipe updated August 6, 2026, the vLLM Kimi K3 release guide, and current vLLM reproducibility and troubleshooting documentation.

The weekly action plan

Treat reproduction as a controlled schedule, not as a sequence of hopeful fixes.

This week, our recommendation is to create one frozen baseline before attempting any repair. If the official compatibility baseline still produces an unstable failure, move the reproduction to an isolated environment rather than continuing to edit the shared cluster.

The failure scene before diagnosis

When Kimi K3 produces a different error after every restart, the problem is usually not that all errors are unrelated. The diagnostic chain has been damaged by simultaneous changes.

A driver upgrade can alter CUDA initialization, peer access, NCCL behavior, and the libraries visible to the container. A new image can change vLLM, PyTorch, FlashInfer, startup defaults, and compiled kernels at the same time. Lowering concurrency or max-model-len can hide an allocation problem without proving that the original environment was compatible. Enabling prefix caching can add another memory and scheduler path before the basic engine has been shown to start correctly.

That creates three hidden costs:

• Causal ambiguity: the first failure is replaced by a later failure, so the team cannot tell which change helped.
• Rollback debt: package histories do not capture host-library selection, container digests, environment variables, or GPU assignment.
• Shared-cluster noise: another process, stale worker, or changed device map can make an identical command appear non-identical.

The first target is not “a working server.” The first target is the same failure at the same stage under the same inputs.

The frozen failure record

Before changing anything, create a directory for one reproduction attempt. Do not overwrite it when the next attempt looks more promising.

mkdir -p repro-2026-08-09/run-001
script -a repro-2026-08-09/run-001/terminal.log

Capture the host and container facts separately. A package list alone is insufficient because the process may load a different driver or library path at runtime.

date -Is
uname -a
nvidia-smi
docker image inspect vllm/vllm-openai:kimi-k3
docker run --rm --gpus all vllm/vllm-openai:kimi-k3 \
  bash -lc 'python -V; python -c "import torch; print(torch.__version__); print(torch.version.cuda); print(torch.cuda.device_count())"'

Save the complete startup command, including environment variables, mounted paths, visible devices, network flags, cache flags, and any shell-generated values. Save the first exception, not only the final process exit message. In a distributed launch, timestamp the first failure from every node.

The record should answer five questions:

1. Which host and GPU devices ran the process?
2. Which image digest and vLLM build were used?
3. Which command and environment variables started the server?
4. What was the first failing stage?
5. Which request or startup option was active at that moment?

The official compatibility boundary

The current official recipe states that the dedicated Kimi K3 container is a CUDA 13, or cu130, build only. It also states that NVIDIA hosts require an r580-or-newer driver. The same page says there is no cu129 tag for that image and explains that a CUDA 12.9 host requires either a driver upgrade or a self-built vLLM path. These are compatibility boundaries, not tuning suggestions. (recipes.vllm.ai)

The recipe lists at least 8 GB300 GPUs for the NVIDIA path and identifies multi-node deployment for real production traffic. Those are official recipe statements and should be checked against the page at the time of testing, not copied from an old internal runbook. (recipes.vllm.ai)

The vLLM release guide also presents an 8-GPU quick-start command and says that the current Docker images are the usable deployment path because of their pre-release dependencies. Keep that distinction clear: a locally installed package may be useful for investigation, but it is not automatically equivalent to the supported Kimi K3 image. (vllm.ai)

The baseline configuration score

Use this score to decide whether the environment is ready for model-parameter investigation. It is not a performance score.

Decision rule: 12–14 points means the team can begin controlled reproduction. 8–11 points means collect missing evidence before tuning. 0–7 points means the run is not a valid diagnostic baseline.

This scoring system does not claim that a high score guarantees a successful launch. It only measures whether the failure can be compared with another run.

The minimal startup command

Remove anything that belongs to the production application rather than the model engine:

• Agent tool definitions and tool-call adapters
• Gateway middleware
• Load-test clients
• Automatic retries
• Long-context defaults
• Speculative decoding
• Multi-node transport overrides
• Prefix-cache retention policies
• Optional structured-output settings
• Application-specific environment variables

Begin with the smallest command that can establish device discovery, weight loading, engine initialization, and API readiness. The exact command must match the image and recipe under test. The official release guide shows the Kimi K3 service pattern with tensor parallelism, remote code trust, fast safetensors loading, explicit prefix caching, and Kimi-specific parsers, but for diagnosis those optional components should be introduced in layers rather than all at once. (vllm.ai)

For a first launch, keep a run sheet like this:

If the first failure occurs during CUDA initialization, device discovery, or NCCL setup, do not jump to OOM or prefix-cache analysis. A later allocation message can be a consequence of incomplete initialization.

For CUDA-side diagnosis, use temporary logging only. Current vLLM troubleshooting guidance documents VLLM_LOGGING_LEVEL=DEBUG, CUDA_LAUNCH_BLOCKING=1, NCCL_DEBUG=TRACE, and VLLM_TRACE_FUNCTION=1; the last option can impose a severe slowdown and should not remain enabled after the evidence is collected. (docs.vllm.ai)

The single-request checkpoint

Once the service reaches readiness, separate startup failure from runtime failure. Send one fixed request through one fixed client path.

Keep these values constant:

• Exact model identifier
• Exact system and user messages
• Fixed maximum output length
• Fixed sampling settings
• Fixed image or multimodal input, if used
• Fixed API route
• Fixed timeout
• One request at a time

Do not use a live Agent workflow as the first request. Tool schemas, retries, streaming, and middleware can add a second failure domain. Use a saved request body and compare the complete response status, server log, memory trace, and process exit state.

For output reproducibility, remember that online serving is not fully deterministic because scheduling can vary. The official reproducibility documentation says vLLM cannot guarantee identical online results in the same way as offline execution, and reproducibility still depends on using the same hardware and vLLM version. Batch invariance is available for testing but remains under active development. (docs.vllm.ai)

That means a stable reproduction does not require identical generated text in every online run. It requires a stable failure stage, stable exit behavior, and a clearly defined request outcome.

The prefix caching checkpoint

Prefix caching should not be part of the first compatibility test. It should be the first optional feature added after one fixed request succeeds.

The current Kimi K3 recipe says prefix caching must be explicitly enabled. The release guide shows --enable-prefix-caching in its quick-start command, while the recipe explains the hybrid cache behavior used by Kimi K3. (recipes.vllm.ai)

Run the cache test as a separate experiment:

1. Start from the known-good single-request command.
2. Add only --enable-prefix-caching.
3. Send the same request twice.
4. Record cache statistics or relevant debug logs.
5. Repeat with the same process, not a new image.
6. Remove the flag if the failure stage changes.
7. Only after that, test a repeated shared prefix.

The expected question is not “did the second response feel faster?” The expected questions are narrower: did the server remain stable, did the cache path activate, and did memory behavior change? A cache miss is not automatically a bug. A startup crash after adding the flag is evidence that the new feature path needs its own investigation.

The OOM and CUDA separation

A Kimi K3 OOM should be classified by timing.

This ordering prevents a common mistake: reducing max-model-len until the process survives, then declaring the CUDA problem solved. If the original failure happened before weights were loaded, the lower context limit did not address the root cause.

For specialized CUDA, OOM, cache, or communication remediation, keep each investigation separate. A dedicated Kimi K3 troubleshooting guide can carry the operational follow-up without turning this reproduction article into a collection of unrelated fixes.

The one-variable recovery loop

After the single request is stable, add production behavior in this order:

1. Prefix caching
2. Longer context
3. Higher output limit
4. Two concurrent requests
5. Higher concurrency
6. Tool calling or structured output
7. Multi-node transport
8. Agent gateway integration

For every step, write down the input condition, the only changed variable, the expected observation, and the rollback command. If the error changes from an OOM to an NCCL failure, do not continue forward. Return to the previous stable baseline and confirm that removing the variable restores the earlier state.

The vLLM recipe includes additional topology-specific settings for RDMA, NVLink, MNNVL, and cross-node communication. Those settings should be added only when the single-node or minimal topology is already understood, because they introduce network and collective-communication variables that can mask model or runtime failures. (recipes.vllm.ai)

The reproduction handoff package

A useful handoff is smaller than a full cluster dump but more precise than a screenshot. Include:

• Host driver output
• Container image digest
• vLLM and runtime versions
• GPU count, names, and visibility
• Full startup command
• Relevant environment variables
• Model revision or local weight identifier
• Fixed request body
• First exception with timestamp
• Stage classification
• One-variable change table
• Result after applying the rollback
• Whether the issue reproduces on a second clean run

Use the JexMac English resource center as the neutral starting point for documenting the surrounding environment and operational assumptions. If the team is comparing temporary test capacity with existing infrastructure, keep the reproduction package independent from the purchasing decision.

The isolation decision

Move to an isolated test environment when at least one of these conditions is true:

• The production host cannot remain on one driver version.
• The image is rebuilt between runs.
• Other jobs change GPU visibility or memory availability.
• A shared scheduler injects environment variables.
• The first error changes without a recorded code or configuration change.
• The team cannot stop automatic retries or health-check restarts.
• The current cluster must continue serving users during diagnosis.

An isolated environment does not prove that production is compatible. It creates a clean comparison point. The correct sequence is to reproduce once in isolation, compare the environment fingerprint, then identify which production variable prevents the same result.

FAQ

Fixed variables for changing startup failures

Capture the host driver, container digest, vLLM version, GPU visibility, full command, environment variables, first exception, request payload, and change order. Package installation records alone are not enough. The goal is to let another engineer reproduce the same stage with the same image and request.

OOM or earlier CUDA failure

Use the timeline. A failure during device discovery, CUDA context creation, NCCL setup, or engine initialization takes priority over a later OOM line. Prove that the smallest supported launch is stable before changing context length, output limits, or concurrency.

Verifying a driver upgrade

Check the driver on the host and GPU visibility inside the exact container. Record the runtime libraries and image digest as well. An upgraded package is not proof that the running process uses the new driver or that an old compatibility path has disappeared.

Adding prefix caching

Add it after the minimal launch and one fixed request pass. The official Kimi K3 recipe requires explicit activation, so test the flag as one controlled change. Compare startup, repeated-request behavior, cache evidence, and memory logs before adding context or concurrency.

Leaving the original cluster

Move when the cluster cannot freeze software, isolate devices, preserve logs, or prevent interference from other tasks. Establish one result on isolated capacity, then compare fingerprints before modifying production. This is faster than repeatedly changing a shared environment with no stable control run.

Current cluster versus isolated Mac testing capacity

A shared CUDA cluster remains the right choice when the team needs the exact supported GPU topology, high-memory distributed execution, or a production-like network fabric. It is a poor long-term place for basic reproduction when driver changes require coordination, other jobs alter the device state, and every test competes with active workloads.

A Mac environment is not a replacement for the official Kimi K3 CUDA serving topology. It does not remove the need for the supported NVIDIA or AMD path when validating the model itself. Its value is different: an isolated, reversible environment for client integration, request shaping, API behavior, log packaging, and control-plane testing before scarce GPU time is consumed.

When the existing setup keeps changing under the investigation, we recommend preparing the minimal command and environment fingerprint from this article, then using JexMac’s available plans to run the surrounding integration work in a separate, controlled environment. That will not fix a CUDA driver mismatch, but it can prevent a shared cluster from becoming the only place where every variable changes at once.

FAQ