Qwen3.8-Flash-Next on a DGX Spark: 23.5 tok/s

Qwen3.8-Flash-Next shipped on 26 August 2026 — the first public preview of the architecture Qwen4 will be built on. It is a 125B model with 6B active parameters, which is exactly the shape a DGX Spark is supposed to like.

I checked whether it would run on ours and concluded it would not. That was wrong, and the way it was wrong is the most useful thing in this post.

The mistake

The official FP8 checkpoint is 172.78 GiB. This machine has 121 GiB of unified memory. Qwen’s own vLLM guidance says TP2 on GB300 is the minimum validated configuration, with TP4 recommended — two to four datacentre GPUs.

Both facts are true. Neither answers the question I was asking.

They describe the unquantised checkpoint under vLLM. Reading a vendor’s datacentre deployment note as the model’s hardware requirement is precisely the error this site exists to correct, and I made it before catching it. The question was never “does the FP8 checkpoint fit” — it was “can this machine run this model”, and those have different answers.

What actually fits

Unsloth’s dynamic GGUF quantisations, and every one of them clears 121 GiB:

(Sizes and retention figures are Unsloth’s, not measured here.)

The choice between them is not free, and on this machine it is not really a quality decision at all. This Spark runs production services — embeddings, a fast generation tier, TTS — that hold about 28 GiB permanently. That leaves ~93 GiB. UD-Q4_K_XL at 103.7 GiB would mean taking those down; UD-Q3_K_XL at 83.8 GiB coexists with them. I took the one that coexists, and the three points of retained accuracy are the price of not interrupting anything.

Getting it to load at all

The official llama.cpp CUDA image has an arm64 build and detects the GB10 correctly. It also refuses the model:

error loading model: unknown model architecture: 'qwen4exp'

Upstream llama.cpp has not merged support. PR #27742 is open and being actively worked by the llama.cpp team after Unsloth handed the draft over. So the model exists, the quantisations exist, and the runtime to load them does not — yet.

Building from the PR branch works:

git clone https://github.com/unslothai/llama.cpp.git
git fetch origin qwen4exp/qwen3.8-flash-next && git checkout FETCH_HEAD
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=121 \
      -DGGML_CUDA_FA=ON -DLLAMA_CURL=OFF -DCMAKE_BUILD_TYPE=Release
cmake --build build -j 18 --target llama-server

CMAKE_CUDA_ARCHITECTURES=121 is the GB10’s compute capability and is not optional. Two details cost me time and are worth writing down: the PR page refers to unsloth/llama.cpp, but the organisation is unslothai — the first clone fails with a credentials prompt, which looks like a private repo rather than a typo. And the build takes about seven minutes on twenty cores.

The model then loads in 102 seconds and occupies 98 GiB of the machine’s 121.

The numbers

23.5 tok/s on chat_short, 24.2 on code_generation, both with a standard deviation near 0.1. It passed the coherence gate 3/3 and wrote a correct merge function for two sorted lists. It works.

It is also slower than Nemotron 3.5 by a factor of three, while carrying twice the active parameters. Six billion active against Nemotron’s three; 23.5 tok/s against 78.35 with both drafters off.

What I cannot tell you

Why it is slow. There are at least three candidate explanations and this run separates none of them:

1. Three-bit quantisation. Q3_K weights need unpacking work per token that NVFP4 does not, and llama.cpp’s kernels for that on Blackwell are new.
2. An unmerged graph. The PR is days old and still landing optimisations — maintainers have been reducing the per-layer-embedding table’s memory residency while this was being written. Nobody has tuned this for SM121.
3. The architecture itself. Possible, and the least likely of the three, but it cannot be ruled out from one run on one backend.

Attributing the gap to any of these would be guessing. The honest statement is that a 6B-active model measured 23.5 tok/s here under a three-day-old implementation, and that this is not the number the architecture will eventually produce on this hardware.

Whether it survives real load. Issue #27780 reports llama-server aborting with SIGABRT during sustained multi-request load — specifically on SM121, which is this exact chip. The reporter hits it deterministically at a particular batch shape, and single or short requests are fine. My run was twelve sequential requests and never came close to that profile, so I did not reproduce it and cannot say it is fixed. There is a second open report of garbage output on SM110 with more than eight layers offloaded.

This is why there is no MMLU-Pro sweep in this post. A subject sweep is 200 questions at eight-way concurrency, which is the load profile that report describes. Running it now would produce either a crash or numbers I could not vouch for.

Anything about quality. The 90.4% retention figure above is Unsloth’s, not ours. What a three-bit quantisation costs this particular model on real benchmarks is unmeasured, and it is a genuine question — the model stores a large fraction of its knowledge in a 51B-parameter N-gram lookup table, and how that table survives aggressive quantisation is not something you can assume from experience with dense weights.

The context irony

Qwen3.8-Flash-Next has a 262,144-token native context, extensible to a million with YaRN. On this machine I gave it 8,192.

Not because of a limitation in the model, but because 83.8 GiB of weights against 92.6 GiB of free memory leaves under nine gigabytes for everything else. The headline context length is a property of the architecture; the context you actually get is a property of your machine, and on a box like this one the weights win.

That is the trade the whole exercise illustrates. A 121 GiB desk-side machine can hold a model whose FP8 checkpoint needs four datacentre GPUs — but it holds it in three bits, at a third of the speed of a model half its size, with a context window a thirtieth of what the model supports.

Total run time

Five minutes of measurement, against a day of build work.

The ratio is the story of this post. The model needs an unmerged pull request to load at all, and once it loads the benchmark is over before you have finished reading the log. Nothing in this figure reflects what it took to get there.

Would I use it

Not yet, and not for anything that matters. It is a research preview running on an unmerged pull request with an open crash report naming this chip. That is three reasons to wait, and none of them are about the model.

What it demonstrates is that the frontier’s newest architecture will run on hardware you can put on a desk, within a day of release, if you are willing to compile it yourself. That was not obviously true a year ago.

I will re-measure when the PR merges. If the gap to Nemotron closes, the explanation was the implementation. If it does not, the explanation is more interesting.

Source data: 028ad1c3369d