Three models on a DGX Spark: fewest bytes still loses

Three models fit comfortably on a DGX Spark’s 121 GB of unified memory, and they represent three different bets about where performance comes from: a dense 27B at 8 bits, a 35B mixture-of-experts at 16 bits with about 3B active per token, and a 117B mixture-of-experts at 4 bits with about 5.1B active.

Measured on the same machine, same harness, same 32k context, the ranking is not what parameter counts suggest. It is also not what the standard mental model for memory-bound decode suggests — and chasing down why turned out to be the interesting half: one serving flag is worth 1.67×, and turning it off reverses the result.

What is being compared

All three served by vLLM 0.19.0 on the same GB10, one at a time.

The metric is end-to-end throughput: total tokens generated divided by total wall time. It is the only figure available for all three, because vLLM buffers gpt-oss’s stream and makes token-level timings meaningless there. It includes prefill, so it is a floor on decode rather than a substitute for it.

Results

The 35B-A3B wins every scenario. gpt-oss is second everywhere, and roughly 2.1× faster than the 27B throughout — a model with four times the parameters, running twice as fast, consistently.

The 27B result needs no explanation

Decode on this hardware is memory-bandwidth-bound. Producing one token means reading the active weights out of memory, and the GB10 has roughly 273 GB/s to work with. So count bytes:

A dense model pays for every parameter on every token. The 27B moves ten times the data gpt-oss does, and measures roughly half the speed. That is the whole story there, and it is the reason “how many parameters” is a poor proxy for “how fast”: what matters is how many of them you touch, and how wide each one is.

Why the fewest bytes still lost

Read that table again. gpt-oss reads the fewest bytes per token of the three — 2.6 GB against the 35B’s 6 GB. On a purely bandwidth-bound view it should be roughly 2.3× faster than the 35B.

It is 26% slower.

So the model that explains the 27B cleanly gives the wrong answer for the two MoEs. Something is adding time that bytes-per-token does not account for.

The answer is in the configuration table: both Qwen models run MTP speculative decoding, and gpt-oss does not. Speculative decoding drafts several tokens per forward pass and has the full model verify them; when drafts are accepted, you get multiple tokens for one weight read. It is precisely a way to beat the bandwidth bound.

I published this as a hypothesis and then tested it. Serving the 35B with the identical configuration minus --speculative-config:

Without speculative decoding the 35B drops to 30.1 tok/s — below gpt-oss-120b’s 37.0. The ranking inverts, and it inverts to exactly the order bytes-per-token predicts.

So the bandwidth model was not wrong. It was describing the hardware correctly, and one of the three models was being given a 1.67× head start by a serving flag. Remove the flag and the physics reassert themselves: the model that reads 2.6 GB per token beats the one that reads 6 GB.

The generalisable finding: a decoding technique outweighed a 2.3× advantage in bytes moved per token. If you are comparing models on throughput, the serving configuration can matter more than the architecture — and a benchmark that does not state its flags is not telling you much.

It also reframes what gpt-oss is. It is not architecturally slow at inference; it is unaccelerated.

The obvious follow-up is whether you can simply turn speculative decoding on for it, and the answer is no: the checkpoint contains no draft heads. Zero of its 687 tensors match anything MTP-shaped, and GptOssForCausalLM exposes no speculative configuration. Qwen ships the draft heads inside the model weights; OpenAI did not. So this is not a flag you can flip — the 1.67× is available to one of these models and not the other, and that is a property of what was shipped rather than of how I configured it.

(Speculation without built-in heads is possible — n-gram, or a separate draft model — but both are different techniques with different trade-offs, and neither is what the Qwen models are doing here.)

Prefill, measured separately

Prefill needed its own method, because deriving it from time-to-first-token requires a server that streams token by token, which gpt-oss does not. Holding output length fixed and varying prompt length isolates it from total wall time instead — the slope of that line is the prefill rate.

Same ordering as decode, which is why the ranking holds across every prompt length rather than crossing over somewhere. Nobody wins at long context by having unusually fast prefill.

That method agreed with the streaming measurement to within 2% on the 27B and 6% on the 35B, which is the only reason I trust it on gpt-oss, where there is nothing to check it against.

Updated 9 August: the gpt-oss figure first published here was ~2,500 tok/s, derived by hand rather than measured, because the measuring tool had the same prefix-caching bug described in the gpt-oss post — it repeated an identical prompt at each length and reported cache lookups, returning an impossible 201,839 tok/s with an r² of 0.68. Re-measured with a unique prefix per request, the answer is 3,283 tok/s at r² 0.9960. The ordering, and every conclusion drawn from it, is unchanged.

Cold starts

None of these are always-on. Time from “a client asks” to “the endpoint answers”, measured with an empty GPU:

Seven to nine and a half minutes. Note the 27B — 27 GB of FP8 weights — loads slower than the 35B’s 70 GB of bf16. Weight size is not the bottleneck; engine initialisation is. I still cannot account for that one.

What I would run

The 35B-A3B, on this hardware, for general work. It is fastest at every prompt length by a comfortable margin, and it loads quickest.

gpt-oss-120b if you want the largest model that fits and can accept 26% less throughput. It is a 117B model running at 37 tok/s on a desk-side box, which is a genuinely surprising thing to be able to say — and note that its deficit is a serving flag rather than the model. It beats the 35B outright once speculative decoding is off on both.

The 27B only if you have measured it winning on your specific task. It is the slowest of the three by a factor of two, and the dense architecture is why.

Total run time

Eighteen minutes to rank three models on speed.

That is the number worth holding next to the quality comparison, which needed 36 hours to rank the same models on accuracy — roughly 120 times as long. Speed is nearly free to measure and quality is not, which is why so much local-LLM writing reports the first confidently and asserts the second.

What this does not tell you

Quality. These are throughput numbers and nothing else. I have MMLU-Pro scores for the two Qwen models — 82.1% and 81.4%, statistically indistinguishable — and none at all for gpt-oss, so a three-way quality comparison would be two-thirds of a table. That measurement is running now and is the next post.

Until then, the honest reading of this article is “which of these is fastest”, not “which of these you should run”. Those are different questions, and the first one only decides the second when the answers to the second are equal.

Concurrency. Single-stream throughout: one request at a time. Batching changes the picture and is not measured here.

Methodology

Five measured repetitions per scenario after a discarded warmup, greedy decoding, medians reported, run under a coordination gate that refuses to measure while another project is using the GPU.

Each repetition uses a unique prompt prefix. That detail is not incidental: vLLM enables automatic prefix caching, and an earlier version of these measurements sent an identical prompt every time. Every recorded repetition was a cache hit, and the long-prompt numbers for gpt-oss came out up to 81% too high before I caught it. That correction is documented in the gpt-oss post. The Qwen models were unaffected — speculative decoding disables prefix caching, which is a coincidence I only understood after measuring both ways.

All three runs here were taken after that fix, with the same harness on the same machine on the same day.

Source data: b43438259dc696f13c64aa0699ee050fe7f2