Qwen3.8-27B on a DGX Spark: half the tokens, 2.8 down

Muse Glimmer took most of a day to serve at all — a pre-release image built from an unmerged pull request, and a 47 GB checkpoint shard that would not stage into 128 GB of memory.

Qwen3.8-27B took about eight minutes. It is the easiest model this machine has ever been handed, and the only thing that will trip you up is a flag the model’s own config file gives you no reason to set.

The thing that will trip you up

Serve it the obvious way and it runs at 7.85 tokens per second. Serve it with one extra flag and it runs at 16.13.

--speculative-config '{"method":"mtp","num_speculative_tokens":2}'

Qwen3.8-27B ships a multi-token-prediction drafter in the checkpoint. Nothing in config.json says so — there is no num_nextn_predict_layers key, no speculative block, nothing. The only evidence is in the weight names:

mtp.fc
mtp.layers
mtp.norm
mtp.pre_fc_norm_embedding
mtp.pre_fc_norm_hidden

If you go looking for a reason to enable speculative decoding by reading the config, you will not find one, and you will leave half the model’s throughput on the floor. I did exactly that on the first run.

Across the suite, MTP is worth 1.5× to 2.1×:

The pattern is the familiar one: speculation buys decode and does nothing for prefill, so the gain tracks how much of each scenario is generation. code_generation is nearly all decode and gains 2.09×; summarization is 16k in and 128 out and gains 1.52×.

How to check any model for this in one command

The general lesson is worth more than the flag. A checkpoint can ship a drafter without advertising it. Look at the weight names, not the config:

python3 -c "
import json, glob
idx = glob.glob('<snapshot>/*.index.json')[0]
w = json.load(open(idx))['weight_map']
print(sorted({k.split('.')[0] + '.' + k.split('.')[1]
              for k in w if any(s in k.lower() for s in ('mtp', 'nextn', 'draft'))}))
"

Empty output means there is nothing to turn on. Anything else means you are currently running slower than you have to.

What did not trip me up

Three things that cost real time on other models and cost nothing here.

No custom image. Qwen3.8-27B declares Qwen3_5ForConditionalGeneration — byte-identical to the architecture Qwen3.6-27B already uses — and the stock nvcr.io/nvidia/vllm:26.04-py3 registers it. Ask the image what it supports before you go looking for a version number:

docker run --rm --entrypoint python3 <image> -c "
from vllm.model_executor.models.registry import ModelRegistry as R
print('Qwen3_5ForConditionalGeneration' in set(R.get_supported_archs()))"

No staging wall. Largest shard is 6.0 GB at FP8 and 4.0 GB at BF16. vLLM stages one shard at a time, so the number that matters is the largest shard, not the 30.9 GB total. Muse Glimmer’s single 47 GB shard is what forced a re-sharding script; nothing here comes close.

No gate, no licence surprise. Apache 2.0, ungated.

Take FP8, and take it for a specific reason

Both Qwen/Qwen3.8-27B (55.6 GB, BF16) and Qwen/Qwen3.8-27B-FP8 (30.9 GB) fit this machine comfortably. I benchmarked FP8 because Qwen3.6-27B was benchmarked at FP8. Running the successor at BF16 would confound a version change with a precision change, and the resulting comparison would measure nothing you could name.

That is the whole reason to care which one you pick.

The next trap: it emits </think> and nothing parses it

Ask it a question and the raw response looks like this:

We need answer "What is the capital of France?..." Final only city name.
</think>

Paris

The reasoning is in content, terminated by a bare </think> with no opening tag. vLLM’s own parser documentation explains why: “Starting with Qwen3.5, the chat template places <think> in the prompt so only </think> appears in the generated output.” The matching parser is --reasoning-parser qwen3, which moves the reasoning into reasoning_content and leaves content as the answer.

Do not turn it on for a benchmark. I measured one subject both ways before committing a fourteen-subject sweep to it:

An 18.5-point collapse on near-identical generation volume — the model behaved the same, the harness stopped being able to see it. The sample logs give the mechanism outright: with the parser, 23% of responses arrive with content as the empty string, because the parser moved the entire response into reasoning_content. lm-eval reads content, finds nothing, and scores a correct answer as wrong.

The general shape is worth keeping: a reasoning parser is the less forgiving setting for regex answer extraction. Without one, a matcher that takes the last answer is X anywhere in the response can still find it; with one, a final answer that does not restate the letter becomes a zero.

Had I trusted the setting instead of testing it, this post would have reported Qwen3.8 at 72.5% on biology and I would have written up a model collapse that never happened. Without the parser it scores 91.5 — a shade above its predecessor.

TTFT stops meaning anything under speculation

With MTP on, the harness records time-to-first-token as null rather than a number. That is deliberate: speculative decoding emits several tokens per stream delta, so the first delta is not the first token and any TTFT computed from it is a fiction. The baseline run measured 176–182 ms; the MTP run reports nothing.

If you are comparing TTFT across a speculative and a non-speculative configuration, you are comparing two different quantities.

So is it better than Qwen3.6-27B?

No — and the way it is worse is more interesting than the fact.

On quality it is 2.79 points behind, 79.75% against 82.54% across the same 2,800 MMLU-Pro questions. At 2,800 per model the 95% interval on that difference is ±2.11, so the gap is real rather than sampling noise.

The aggregate hides a split that is almost perfectly clean:

Qwen3.8 gains on technical subjects and loses badly on knowledge-and-judgement ones. Philosophy −13.0, law −12.0, “other” −10.0 against chemistry +4.5 and engineering +4.0.

Those three biggest losses are not random. The per-subject comparison across four models found that law, history and philosophy carry nearly all the signal that distinguishes models at all — the technical subjects cannot tell them apart. Qwen3.8 gave up ground precisely where models are most separable.

Is it the model, or its FP8 build?

The obvious objection to everything above: both models were measured at FP8. If Qwen3.8’s quantisation happens to be lossier than Qwen3.6’s, the deficit is an artifact and this post is wrong.

So I ran the unquantised weights — Qwen/Qwen3.8-27B, 55.6 GB, which vLLM serves as bf16 — on the three subjects carrying the entire deficit.

Full precision recovers nothing. The mean difference is +0.67 points against a 95% interval of ±5.19, and the three deltas scatter in both directions — noise, not a suppressed effect. Against Qwen3.6 the gap is still 11 points at full precision.

The FP8 build is fine. The model is genuinely weaker here.

That test also answers a question worth having on its own: FP8 costs no measurable accuracy and runs 1.61× faster — 95.2 tok/s against 59.0 for bf16, on identical questions. If you were wondering whether to pay the memory and the throughput for full precision on this model, the answer is no.

It answers twice as fast, which is not the same as being faster

This is the part a tokens-per-second table gets backwards.

Every number in the first row is correct and the conclusion it invites is wrong. Tokens per second measures how fast the machine emits text. No workload is “emit text” — an agent step, a grading call, a chat turn all end when the answer is complete. On that measure Qwen3.8 is not 8% slower, it is nearly twice as fast, because it needs half as many tokens to get there.

The wall-clock row is not derived: it is the measured runtime of the identical sweep on the identical machine.

Does thinking less cost it the accuracy?

Tempting, and not established. Across the fourteen subjects, the correlation between how much Qwen3.8 cut its token count and how much accuracy it lost is r = +0.476 — below the ±0.53 significance threshold at n=14.

The two subjects where it gains are the two where it cuts tokens least (engineering 1.46×, chemistry 2.23×), which is suggestive. But philosophy loses 13 points at a 1.69× cut while economics loses nothing at 2.84×, and that is the wrong shape for a clean mechanism.

So: a plausible hypothesis, not a finding. Separating it would need the same question set at matched generation lengths, which is a different experiment.

Which one should you run

Qwen3.6-27B if your work is knowledge- or judgement-shaped — law, philosophy, general reasoning. The gap there is 10 to 13 points, far outside noise, and no amount of speed compensates.

Qwen3.8-27B if your work is technical and latency matters. It is level or slightly ahead on chemistry, engineering, maths and physics, and it gets there in half the wall clock. For an agent loop making many calls where each one ends at a complete answer, halving time-to-completion is worth more than 0.4 points of accuracy.

Qwen3.8-27B also if you need vision — it reads images and its predecessor does not, which is the one thing here that is not a trade-off.

Total run time

About thirteen hours, most of it the fourteen-subject FP8 sweep.

The headline claim of this post is that Qwen3.8 answers in half the wall clock, and this is where that shows up as something other than a table: 8.1 hours for 2,800 questions against Qwen3.6-27B’s 14.9 for the same set. Half the tokens, roughly half the time.

The two parser arms are cheap — 27 minutes each, one subject — because that question only needed one subject to answer. The bf16 check is three subjects, run to confirm the FP8 quantisation was not responsible for the score difference.

Setup

• NVIDIA GB10, 121 GB unified memory
• vLLM 0.19.0, nvcr.io/nvidia/vllm:26.04-py3, 32k context, --gpu-memory-utilization 0.65
• Qwen/Qwen3.8-27B-FP8, served under the name Qwen/Qwen3.8-27B
• Both runs passed the harness’s 3/3 coherence probes before any timing was recorded
• Single stream, one discarded warmup, five measured repetitions, medians reported
• Speed run records: 038e8db7e5c0 (no drafter), a017ae89ef4e (MTP ×2), 2f908ad0ef76 (Qwen3.6-27B, published). Both Qwen3.8 runs were re-measured with --serving-command so the serving configuration is recorded; the originals (09d27b786af4, 28ea375d74ca) reproduced within 0.5% and are kept in the repository
• Quality: 14 subjects × 200 questions = 2,800 per model, 5-shot CoT, greedy, mmlu_pro_fair. Result files in data/subject-sweep/
• No --reasoning-parser, deliberately — see above. The four other models on this machine were measured the same way

Source data: 038e8db7e5c0a017ae89ef4e2f908ad0ef76