Local LLM Labs

Running gpt-oss-120b on a DGX Spark: four things that will trip you up

updated August 9, 2026dgx-spark

Correction, 9 August 2026. The long-prompt numbers in the original version of this post were wrong, and so was the conclusion I drew from them. My harness sent an identical prompt for every repetition and discarded only the warmup, so every recorded repetition was served from vLLM’s automatic prefix cache. Prefilling 27,300 tokens took 8.5s on the first request and 0.6s on the next — and only the second kind was being measured. chat_long_context was overstated by 19%, summarization by 81%, and the claim that gpt-oss is “astonishingly flat” across prompt lengths was an artifact of that, not a property of the model. Corrected figures are below with the originals struck through; the withdrawn run is kept in the repository rather than deleted. The two short-prompt scenarios were unaffected. Details at the end.

OpenAI’s gpt-oss-120b is 117B parameters with about 5.1B active per token, natively quantized to MXFP4. That combination makes it one of the largest models that genuinely fits on a DGX Spark’s 121 GB of unified memory, and the obvious question is what it feels like to run.

Short answer: about 37 tokens per second on short prompts, falling to 19.5 by ten thousand tokens of input, while three other models stay loaded alongside it. Getting to that number took four detours, none of which are in the quickstarts — and one of them I got wrong the first time and published anyway.

1. The download is 133 GB unless you filter it

The obvious command is the wrong one:

snapshot_download("openai/gpt-oss-120b")   # don't

The repository carries more than the MXFP4 weights. There is a metal/ directory for Apple silicon and an original/ checkpoint, and a bare snapshot_download fetches all of it. Mine passed 133 GB with a 31 GB blob still in flight before I noticed, for a model that needs 57 GB.

snapshot_download(
    "openai/gpt-oss-120b",
    ignore_patterns=["metal/*", "original/*"],
)

What vLLM actually needs is fourteen model-000NN-of-00014.safetensors shards, config.json, model.safetensors.index.json, the tokenizer files and chat_template.jinja. If you already made this mistake, the completed shards are reusable — deleting the .incomplete blobs and the two extra directories recovered 56 GB without re-downloading anything.

2. VLLM_MXFP4_BACKEND=marlin is not optional

This is the one that costs an evening.

On SM121 — which is what the GB10 reports — vLLM’s default CUTLASS FP4 path corrupts the first Harmony token. The symptom is not a crash or an error. The server starts, answers requests, and returns:

{"choices": [{"message": {"content": null}}]}

A model that loads fine and returns nothing looks like a broken download, a bad chat template, or a model that simply does not work on your hardware. It is a backend flag. There is an open vLLM issue describing exactly this on DGX Spark.

Marlin is currently the only MXFP4 backend that produces correct output on this silicon.

3. It does not have to monopolise the machine

Published guidance for gpt-oss-120b on a Spark tends to quote around 115 GB of memory in use, which reads as “this model owns the box”. That is what you get at default utilization, not what you are stuck with.

At --gpu-memory-utilization 0.65 it fits in roughly 76 GB and coexists with everything else I had running — an embedding model, a small chat model, and a TTS service, about 26 GB between them. Total system usage settled at 102 GB of 121, and none of the other services were evicted or degraded. I checked by sending real requests to them, not by watching the port stay open.

The launch that worked:

docker run -d --name vllm-gpt-oss-120b --gpus all --ipc=host --shm-size=16g \
  -p 8006:8006 \
  -e VLLM_MXFP4_BACKEND=marlin \
  -v /opt/vllm/cache:/root/.cache/huggingface \
  nvcr.io/nvidia/vllm:26.04-py3 \
  vllm serve openai/gpt-oss-120b \
    --port 8006 --host 0.0.0.0 \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.65 \
    --max-model-len 32768

--tensor-parallel-size 1 matters: the Spark is a single chip, and anything higher fails at launch.

Cold start was 570 seconds — nine and a half minutes from docker run to the endpoint answering. Budget for that. It is not a hang.

4. Your benchmark will lie about the throughput

This one is worth dwelling on, because the wrong number was extremely convincing.

My harness first reported 198 tok/s on short chat and 772 tok/s on sustained code generation. Those are not slightly optimistic, they are impossible: the GB10 has roughly 273 GB/s of memory bandwidth, and at ~2.5 GB of active weights read per token the hard ceiling is near 100 tok/s.

The cause is that vLLM does not stream this model token by token. Harmony channel parsing buffers output, and I measured 29 tokens arriving in 2 server-sent events. My harness timed the gap between the first and last event — milliseconds — and divided the full token count by it.

Two scenarios also “failed” outright with no content tokens, which was the same bug wearing a different hat.

The fix is to measure something that survives buffering:

end-to-end throughput = completion tokens ÷ total wall time.

It is well defined no matter how a server chunks its stream. It includes prefill, so it is a floor on decode rather than a substitute for it — but it is honest, and it is comparable across servers that behave differently.

Time-to-first-token is unavailable here for the same reason, and I no longer report it for this model. The tell was that “TTFT” measured 3,970 ms for a 256-token output and 16,900 ms for a 1024-token one — a ratio that follows output length, not latency. It was timing when the buffer flushed.

The numbers

Five measured repetitions with a per-repetition nonce, greedy decoding, medians reported. Prompt lengths from 167 to 10,770 tokens.

End-to-end throughput — code_generation428 prompt tokens · median of 5 runs
MachineModeltok/s
NVIDIA DGX Spark (GB10)
NVIDIA GB10
openai/gpt-oss-120b
MXFP4 · vllm
37.0 ±0.2
420-token prompt, 1024-token output. Higher is better.
Scenario Prompt Output End-to-end Originally published
chat_short 167 256 37.2 tok/s 37.8
code_generation 420 1024 37.0 tok/s 37.9
chat_long_context 5,434 256 30.7 tok/s 36.4
summarization 10,770 128 19.5 tok/s 35.3

Standard deviation across repetitions was under 0.2 tok/s.

The two short-prompt scenarios were always right — prefill is negligible there, so caching had nothing to hide. The two long-prompt figures were wrong by 19% and 81%, and the original version of this post drew its main conclusion from them.

The corrected shape is ordinary: throughput falls by about half between a 167-token prompt and a 10,770-token one, because prefilling ten thousand tokens at roughly 2,500 tok/s takes four seconds and the request only generates 128. That is what every model on this box does. gpt-oss is not special here, and the first version of this post said it was.

That ~37 tok/s figure still lands on the lower of the two numbers circulating for this hardware. Published reports say both ~59 and ~37–39 tok/s, which looks like a contradiction and probably is not: end-to-end and decode-only are different quantities, and the gap between them is exactly the sort of thing that goes unstated. If you are comparing your own number to someone else’s, check which one they measured before concluding anything.

What it is like to use

Ten minutes to start, then roughly 37 tokens per second sustained. That is usable for interactive chat — comfortably faster than reading speed — and slow enough that you notice it in an agentic loop where each step waits on the previous one.

The more surprising practical fact is the coexistence. A 117B model sharing a desk-side machine with an embedding service, a small chat model and TTS, all responsive, is not what the memory figures in most write-ups would lead you to expect.

Setup recorded with the run

A comparison against the Qwen3.6 models on this same box is coming, once all three have been measured with the same metric. Comparing an end-to-end number against a decode number would be exactly the mistake this post is about.

What went wrong

Two measurement failures in one post, and I want both on the record.

The one I caught before publishing is described above: vLLM buffers this model’s stream, so timing between server-sent events reported 772 tok/s on hardware whose bandwidth caps it near 100.

The one I published was subtler. My harness runs one warmup, discards it, then measures five repetitions of the same prompt. Discarding the warmup was deliberate — it absorbs weight loading and cache warming. But these prompts are built from a repeated filler sentence, so every one shares a prefix with every other, and vLLM’s automatic prefix caching serves the repetitions from cache. The warmup was the only request that paid real prefill, and it was the one I threw away.

The proof is a shared-prefix versus unique-prefix comparison:

Prompt Shared prefix Unique prefix
~6,900 tokens 2.17s, then 0.52s, 0.50s 2.21s, 2.19s, 2.21s
~27,300 tokens 8.46s, then 0.59s, 0.60s 10.28s, 10.32s, 10.33s

The first request pays; the rest are lookups.

Two things made this hard to notice. The numbers were stable — standard deviation under 0.04 tok/s, because cache hits are very consistent. And they were plausible: a fast model looking flat across prompt lengths is exactly what a good MoE should do.

It also only affected one model. The Qwen models on the same box run MTP speculative decoding, which disables prefix caching, so their measurements were real all along. I was comparing two models under different caching regimes without knowing it — which is worse than a wrong number, because nothing about the output says so.

The fix is a per-repetition nonce at the front of each prompt, so no request can be a cache hit, and a prompt_nonce field in every result file so runs from before and after the fix cannot be silently compared.

The general lesson is the same one as the lm-eval post, which I wrote a day earlier and evidently had not internalised: a benchmark that returns stable, plausible numbers has told you nothing about whether it measured what you think.

Source data: 99ee050fe7f2