Nemotron 3.5 on a DGX Spark: 139 tok/s

NVIDIA’s Nemotron 3.5 Lightning 30B-A3B generates 139 tokens per second on a DGX Spark, in the same configuration every other model on this machine was measured in. The next fastest thing here does 52.

Three things make it worth a post beyond the number. A single flag accounts for nearly all of the speed. It serves a genuine one-million-token context on a 121 GiB machine — not an advertised limit, a measured one. And I spent an evening concluding the model could not run on this hardware at all.

The wrong container

The first attempt failed before a single weight loaded:

Value error, The checkpoint you are trying to load has model type
`nemotron_h` but Transformers does not recognize this architecture. This
could be because of an issue with the checkpoint, or because your version
of Transformers is out of date.

The last clause is the answer, and I read past it. I checked vLLM’s ModelRegistry, found NemotronHForCausalLM present, checked Transformers’ CONFIG_MAPPING_NAMES, found nemotron_h absent, and concluded the model needed a Transformers release that had not shipped yet. That reasoning is sound. The conclusion was wrong, because I ran it against nvcr.io/nvidia/vllm:26.04-py3 — NVIDIA’s own container, which ships vLLM 0.19.0.

vLLM added Nemotron 3.5 support on day zero, in 0.27.1, and vllm/vllm-openai:v0.27.1 publishes a linux/arm64 manifest that runs natively on a GB10. In that image both registries answer yes:

transformers 5.15.0 knows nemotron_h        : True
vllm 0.27.1 has NemotronHForCausalLM        : True

The lesson is not “check the registry” — I did check the registry. It is that a registry check is a statement about the image you ran it in, and an unsupported architecture and a stale container produce the identical error. Worth noting that the vendor’s own container was the stale one and the upstream community build was current.

Setup

Three configurations, same weights, same image, same machine:

• Fleet-matched — 32k context, --gpu-memory-utilization 0.65, no speculative decoding, no prefix caching, no fp8 KV cache. Identical to how the other five models here were measured, which is the only reason the comparison at the end means anything.
• + DSpark — the above plus NVIDIA’s DSpark drafter at depth 3, and the Mamba stochastic-rounding state rollback that drafting requires. One change.
• NVIDIA’s recipe — the published DGX Spark command: DSpark, marlin MoE kernels, fp8 KV cache, prefix caching, a larger batch budget.

The first two set the context window explicitly to 32,768, matching the rest of the fleet. NVIDIA’s command does not pass --max-model-len at all, so that configuration takes whatever vLLM resolves from the checkpoint — which turns out to be 1,048,576. More on that below.

Two departures from NVIDIA’s published command are recorded in the run: the default --gpu-memory-utilization of 0.92 asks for 111.95 GiB on a machine with about 99 GiB free and refuses to start, so it is set to 0.8; and --compilation_config.cudagraph_capture_sizes is left at its default rather than the published explicit list, because guessing that list would have made it a different experiment.

All three passed the coherence gate 3/3 before any timing was recorded.

The exact commands

Read out of the run records, not retyped — each is the command that produced the numbers below. Substitute your own weights cache path for /opt/vllm/cache.

docker run -d \
  --name vllm-nemo-b2 \
  --gpus all \
  --ipc=host \
  --shm-size=16g -p 8016:8016 -v /opt/vllm/cache:/root/.cache/huggingface vllm/vllm-openai:v0.27.1 \
  --model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
  --served-model-name nvidia/Nemotron-3.5-Lightning-30B-A3B \
  --port 8016 \
  --host 0.0.0.0 \
  --trust-remote-code \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.65 \
  --mamba-backend flashinfer \
  --mamba-cache-mode align

docker run -d \
  --name vllm-nemo-c2 \
  --gpus all \
  --ipc=host \
  --shm-size=16g -p 8016:8016 -v /opt/vllm/cache:/root/.cache/huggingface vllm/vllm-openai:v0.27.1 \
  --model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
  --served-model-name nvidia/Nemotron-3.5-Lightning-30B-A3B \
  --port 8016 \
  --host 0.0.0.0 \
  --trust-remote-code \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.65 \
  --mamba-backend flashinfer \
  --mamba-cache-mode align \
  --mamba-ssm-cache-dtype float16 \
  --enable-mamba-cache-stochastic-rounding \
  --mamba-cache-philox-rounds 5 \
  --speculative_config.method dspark \
  --speculative_config.model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark \
  --speculative_config.num_speculative_tokens 3

docker run -d \
  --name vllm-nemo-a2 \
  --gpus all \
  --ipc=host \
  --shm-size=16g -p 8016:8016 -v /opt/vllm/cache:/root/.cache/huggingface vllm/vllm-openai:v0.27.1 \
  --model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
  --served-model-name nvidia/Nemotron-3.5-Lightning-30B-A3B \
  --port 8016 \
  --host 0.0.0.0 \
  --trust-remote-code \
  --gpu-memory-utilization 0.8 \
  --moe-backend marlin \
  --kv-cache-dtype fp8 \
  --max-num-batched-tokens 16384 \
  --enable-prefix-caching \
  --mamba-backend flashinfer \
  --mamba-ssm-cache-dtype float16 \
  --mamba-cache-mode align \
  --enable-mamba-cache-stochastic-rounding \
  --mamba-cache-philox-rounds 5 \
  --speculative_config.method dspark \
  --speculative_config.model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark \
  --speculative_config.num_speculative_tokens 3

Two details in the drafter command are easy to miss and both are load-bearing. --speculative_config.model points at a separate checkpoint — the DSpark draft head ships as its own 1.3 GB repo alongside the weights, and drafting does nothing without it. And --enable-mamba-cache-stochastic-rounding requires --mamba-ssm-cache-dtype float16; vLLM validates the pair and refuses to start if you set one without the other. A hybrid Mamba model has to roll its state back whenever a drafted token is rejected, which is what those flags control. My first attempt at the drafter-only configuration died on exactly that, having carried two of the three flags across from the recipe.

Results

chat_short

A 128-token prompt, 256 out — nearly pure decode, and the closest thing here to what using the model feels like. 78.35 becomes 139.42 with the drafter.

code_generation

512 in, 1024 out: the longest generation in the suite and the shape that matters for agentic use.

chat_long_context

8k in, 256 out — the RAG shape, and the smallest gain of the decode-heavy scenarios for the usual reason: a meaningful share of the wall clock is prefill, and speculation does nothing for prefill.

summarization

16k in, 128 out — prefill-dominated, and the weakest case for drafting at 1.21×.

One flag is most of the story

Speedups over the fleet-matched baseline:

The drafter is worth between 1.21× and 1.78× depending on shape. Marlin MoE kernels, an fp8 KV cache, prefix caching and a 16k batch budget — four changes moving together — add between 2.1% and 6.0% on top of that.

That last figure should not be trusted to be positive. Speculative decoding makes throughput noticeably noisier run to run, because how many drafted tokens get accepted depends on what the model happens to be writing. Measured across repeat runs of identical configurations, the no-drafter baseline reproduced within 0.5% while both drafter configurations moved about 6%. A 2–6% gap between two drafter configurations sits inside that. The honest statement is that the drafter is unambiguous and everything else in the recipe is unresolved at this sample size.

I originally wrote this section around an apparent 7% inversion — the drafter alone beating the full recipe on code_generation. It did not reproduce. That is what a 6% run-to-run spread does to a single-run comparison, and it is why the flags stay in the “unresolved” column rather than the “not worth having” one.

A million-token context, and it is real

NVIDIA’s recipe omits --max-model-len, so vLLM resolved the checkpoint’s full max_position_embeddings — 1,048,576 tokens — and served it. An advertised context is not usually an available one, so it is worth reporting what the engine actually reserved:

Available KV cache memory:  72.03 GiB
GPU KV cache size:          23,338,944 tokens
Maximum concurrency for 1,048,576 tokens per request:  22.26x

Twenty-three million tokens of KV cache, and enough for twenty-two concurrent requests at the full million-token length. That is not a technicality; it is a usable window.

It works because of three things stacking. The model uses two KV heads with a head dimension of 128, where a conventional dense model runs eight or more. The fp8 KV cache halves the remainder. And it is a hybrid: most of its 52 layers are Mamba, whose state is a fixed size per sequence and does not grow with context at all — the measured footprint works out near 3.3 KB per token, so only a handful of layers are paying attention’s price.

For contrast, a dense 27B with eight KV heads at bf16 needs roughly 256 KB per token. A one-million-token window would want about 275 GB of KV cache. That is not a tuning problem; it is twice the machine.

Two models at once: the fast one pays

The Spark has memory to spare at these sizes, so Nemotron and Qwen3.6-27B FP8 were served together and benchmarked alone and simultaneously:

The slow model pays nothing and the fast one pays everything. Qwen3.6-27B is dense, reads ~35 GB of weights per token, and was already saturating the memory bus alone; its demand is inelastic. Nemotron was running on slack, and slack is what a co-tenant takes. The practical version: do not co-host this model with a dense one if you care about its speed.

Full measurement, the two setup failures that precede it, and what it means for running several models at once: Two models at once on a DGX Spark.

Against the rest of the machine

One run per model, each in the fastest configuration measured for it — the Qwen models with their MTP drafters, Muse Glimmer with DFlash at depth 15, Nemotron with DSpark. Same scenario, same harness. Nemotron with its drafter is 2.7× the next fastest — and the model it beats, Qwen3.6-35B-A3B, needs MTP drafting to reach 52.09. The fleet-matched Nemotron run at 78.35 beats it without a drafter at all.

Total run time

Five minutes.

Nemotron is fast enough that the whole scenario set finishes in about the time a slower model takes for one scenario — the Muse Glimmer runs took forty minutes for the same work. Throughput shows up in the benchmark’s own duration before it shows up in the table.

What this does not count is the days spent on the container that would not start, which is most of what this post is actually about.

What this post does not tell you

How good it is. Nemotron scored 72.11% across all fourteen MMLU-Pro subjects, which would place it last in this fleet — and that number is not trustworthy. The benchmark caps generated answers at 3,072 tokens, and Nemotron ran past that cap on 11.89% of questions, getting cut off mid-sentence and scored wrong regardless of what it knew. On engineering it truncated 47% of the time; raising the cap to 8,192 took that subject from 45.0% to 66.5%.

A quality post follows once that is measured properly rather than patched. On this model the obvious reading of a score has been wrong twice now.

Whether it is fast for you. These are single-stream measurements with one request in flight. The contention result above is the beginning of an answer for concurrent serving, not the end of one. And this benchmark deliberately defeats prefix caching with a per-repetition nonce, so every recorded prefill is real work — which is exactly the thing prefix caching exists to avoid in production.

If you are running this on a Spark

Use vllm/vllm-openai:v0.27.1, not the NGC image. The vendor container is several versions behind and fails in a way that looks like the model’s fault.

Turn on DSpark. It is worth 1.21× to 1.78× depending on shape, and it is one flag plus a 1.3 GB drafter checkpoint. The rest of the recipe is worth single digits and this benchmark cannot resolve its sign.

Set --gpu-memory-utilization explicitly. The published command omits it, and the 0.92 default will not start on a Spark that is also driving a desktop.

Do not co-host it with a dense model if you care about its speed. It gives up two thirds of its throughput to a model that gives up nothing.

Judge it on the drafter row. 78.35 tok/s is what this machine does with an A3B at NVFP4 and no drafting. 139 is what the model does configured the way its authors intended.

Source data: 94e2dfaf00fc328836aeb7329cf8c15ac6ba