DeepSeek-V4-Flash on a DGX Spark: 17 tok/s at 3-bit

The standard mental model for mixture-of-experts inference on a memory-bound machine goes like this: only the activated parameters are read per token, so speed tracks active size, not total size. It is why a 30B model with 3B active runs like a small model, and why a DGX Spark — 121 GiB of unified memory at roughly 273 GB/s — is a reasonable machine for sparse models that a 32 GB card cannot hold at all.

DeepSeek-V4-Flash is 290.9B parameters with about 8B active. Nemotron 3.5 is 30B with 3B active. On the same machine, in the same week, the model with 2.7× the active parameters runs at 22% of the speed.

That is a much bigger gap than the mental model allows, and this post is mostly about being careful with it.

Getting it on the machine

DeepSeek publishes under MIT and the BF16 checkpoint is 148.7 GiB, which does not fit. Unsloth’s dynamic GGUF quantisations do:

This Spark permanently runs an embedding model, a fast generation tier and a TTS service, holding about 28 GiB between them. 28 + 95.9 is over 121, so the three-bit quantisation required taking production down for the duration — 25 minutes, in the end.

That was a deliberate choice over the two-bit UD-IQ2_M, which would have coexisted. At two bits you are measuring the quantisation as much as the model. Three bits is also what the Qwen3.8-Flash-Next run used, which makes the two directly comparable — and that comparison turns out to be the interesting part.

The runtime was stock llama.cpp master, built for SM121, deliberately not the patched branch that Qwen model still requires. Architecture support for deepseek4 has been merged for months. It loaded in 96 seconds, sat at 102 GiB resident, and took a 32,768-token context — four times what Qwen managed, because stopping the services bought real headroom.

The numbers

DeepSeek-V4-Flash: 17.16 tok/s on chat_short, 17.45 on code_generation, 9.27 on chat_long_context with its 8,192-token prompt. Standard deviations of 0.01 to 0.44. Coherence gate 3/3, zero failures across fifteen requests.

It works, and it is stable. This is not a broken-server result.

Active parameters do not predict the speed

Line the three up by active parameters and the ordering is exactly wrong:

If activated parameters were the binding constraint, 2.7× the active size should not cost 78% of the throughput. Something else is doing the work, and there were three candidates:

1. Three-bit quantisation — Q3_K and IQ3 need unpacking work per token that NVFP4 does not, on kernels that are new for Blackwell.
2. Backend maturity — llama.cpp’s MoE path is less optimised than vLLM’s.
3. Resident size — both large models sit near the machine’s capacity; Nemotron does not.

The experiment that separates them

Take one model. Hold the architecture, the backend, the build, the flags and the context window fixed. Change only the quantisation.

Qwen3.8-Flash-Next at UD-IQ1_S is 67.6 GiB against 83.8 GiB for UD-Q3_K_XL — same active parameters, same graph, same llama.cpp build, 16.2 GiB less resident.

Resident size down to 0.807×, throughput up 1.27× on short chat and 1.28× on code generation. Pure inverse proportionality predicts 1.240×. The measurement lands just above it.

The confound runs the wrong way, which makes this stronger than a clean test. IQ1’s lookup-table decoding is if anything more expensive per weight than Q3_K, so the quantisation-cost explanation predicts the smaller model should be slower. It was faster anyway. Candidates (1) and (2) are held fixed or pushing against the result, and residency still won.

The relationship also holds across the fleet, which I did not expect:

Resident size varies 4.6× across those rows — two inference engines, three quantisation families. The product varies 1.22×.

On this machine, throughput is governed by how much is resident, not by how much is activated. For sparse models close to the memory ceiling, the usual mental model does not describe what happens.

What I still cannot explain

The mechanism. If every resident byte were read once per token, tok/s × GiB would equal the machine’s memory bandwidth, roughly 254 GiB/s. It is 1645–2015, six to eight times higher. So something reads about a seventh of resident memory per token — but a constant-ish seventh, across models whose activated-to-total ratios are 10%, 4.8% and 2.7%. Those ratios differ by nearly 4× while the product barely moves.

It could be an allocator or paging effect, cache behaviour that scales with footprint, or something particular to unified memory on this part. Naming one would be a guess. The measurement stands without it, and the prediction it makes — that shrinking a model speeds it up roughly in proportion, regardless of how little of it is active — is testable by anyone with one of these machines.

Total run time

Eighteen minutes across three runs — DeepSeek-V4-Flash plus the two Qwen3.8-Flash-Next quantisations it is compared against.

The comparison runs are the expensive part of the claim and the cheap part of the bill. Establishing that throughput tracks resident size rather than active size needed a second model measured at two quantisations, which is three runs instead of one, and still under twenty minutes of machine time.

What else this does not tell you

Nothing about quality. Three-bit weights on a 291B model may hold up well or badly and this post does not know. No MMLU-Pro sweep was run.

Nothing about vLLM. An NVFP4 checkpoint of this model would be the apples-to-apples comparison against the rest of this fleet, and at the sizes DeepSeek publishes it does not fit here.

Nothing about the 1M context. DeepSeek-V4-Flash supports a million tokens natively. It got 32,768, because 95.9 GiB of weights in a 121 GiB machine leaves about 22 GiB for everything else. The long-context figure above is at 8,192 — already a 46% throughput cost against short chat, and the steepest long-context penalty measured on this fleet.

It required the machine to itself. Every other model benchmarked here shares the Spark with production services. This one cannot, at a usable quantisation, and that is a real operational cost that a tokens-per-second number does not capture.

Would I run it

As the everyday model on this machine, no — it is four and a half times slower than Nemotron and cannot coexist with anything else.

As a demonstration that a 291-billion-parameter frontier model will answer coherently on hardware that fits under a desk, at three bits, with a stock runtime and no patches: yes, and that is worth knowing. Two years ago the answer would have been a flat no.

Source data: a6530cf66fb1028ad1c3369daa8df7c97943