Qwen3.8-27B at a 65,536-token cap: the one model that needed it

This is the sixth and last model in a campaign that re-measured every quality number on this site with the answer budget raised twenty-one fold. The first five posts all reached some version of the same conclusion: the 3,072-token default was wrong, and 65,536 was more room than anyone needed.

Qwen3.8-27B is the exception. It is the one model that was still gaining at the top of the range.

The result

79.75 to 83.54, +3.79. Twelve subjects up, one flat, one down.

Engineering leads at +16.0, and the ordering below it tracks one thing:

The three that did not grow are economics, history and psychology. Between them they moved −1.00 on average, which is noise around zero and is what a cap that never bound anything looks like. It is the same split Muse Glimmer showed, and the sixth model in a row to show it.

Where this model breaks the pattern

Every other cap-sensitive model in this campaign had stopped improving well before 65,536. The Qwen3.6 pair were finished by 8,192: their engineering scores went 52.0 → 83.0 → 84.5 → 82.5 and 60.0 → 81.5 → 82.5 → 79.0, flat after the first step. The campaign’s own driver notes that 65,536 was chosen because it is where Nemotron stopped being truncated, and for the others it was insurance rather than points.

Not here:

Engineering runs 64.0 → 67.5 → 75.0 → 80.0. Every step is worth something, including the last one, and the curve has not turned over. The token counts say the same: 1,896 → 3,441 → 5,052 → 7,253 per answer, still climbing at the top.

Law is the counterweight in the same table: 63.5 → 70.5 → 71.0 → 70.5, with answers growing 1,750 → 2,909 → 3,438 → 3,960. All of law’s gain arrives by 8,192 and the rest of the budget buys nothing. So the right cap is not even constant within one model — it is a property of how long that model reasons on that subject, and engineering is the only place in this grid where 65,536 earned its keep.

This is the measurement that justifies the campaign’s cap after the fact. Five models suggested 65,536 was over-provisioned. The sixth would have been under-measured at anything smaller, and there was no way to know which kind a model was without running it.

Law, and a debt from earlier in the campaign

Law fell for both Qwen3.6 models when the cap came off — −5.0 and −2.0 — which was alarming enough to be re-run and shown to be sampling churn: about one answer in ten changes between identical runs, and the flips split evenly.

That result was a negative one. It said the drops were not real; it could not say whether the cap does anything to law at all. Here it does: +7.0, with answers 2.26× longer, and a curve that rises and then flattens exactly as a cap-limited subject should. Law is cap-sensitive on this model, and the earlier drops really were noise rather than something specific about the subject.

Does it change the verdict against Qwen3.6-27B?

The original post on this model concluded that the successor is 2.79 points behind Qwen3.6-27B, losing on knowledge and judgement while gaining slightly on technical subjects. Both models have now been re-measured at the honest cap:

83.54 against 85.96 — the gap closes from 2.79 to 2.42 and does not go away. The shape is unchanged too:

Philosophy is still −13.5, history −5.0, law −3.0. The earlier post’s finding — that this model trades knowledge for technical ability, and gave up ground precisely where models are most separable — survives a measurement in which both models gained three to four points.

What it cost

8.1 hours to 22.3 hours for the same 2,800 questions, at 997 → 2,067 generated tokens per answer. Engineering alone took 6.07 hours of that — one subject consuming more than a quarter of the sweep, and the second most expensive category in the entire campaign.

Throughput fell from 95.3 to 72.0 tok/s on the same machine with the same drafter, which is the expected shape: longer answers mean a larger KV cache per slot at eight-way concurrency.

What this does not tell you

200 questions per subject, and this campaign has measured 8–10% of answers changing between identical runs. Treat single-subject moves under about 5 points as scatter. Engineering’s +16.0, other’s +8.5, law’s +7.0 and chemistry’s +6.5 clear that. History’s −3.5 does not, and its answers grew 15%, so there is no mechanism for the cap to have caused it.

The 3,072 baseline’s serving command was never recorded. That sweep predates the fix that makes the harness record what it actually ran, so the file says serving_command: null. The two arms are comparable on the evidence that survives — the baseline logged speculative-decoding acceptance of 0.77–0.85 and this arm 0.73–0.84, so both had a working drafter of the same kind — but it is weaker than being able to diff two commands, and it is the reason the 65,536 arm’s command was reconstructed rather than copied.

Engineering has not been shown to saturate. This post says the curve was still climbing at 65,536, not that it stops there. A larger cap might buy more. It would also cost more than six hours for one subject, and this campaign chose to stop.

Nothing here is about speed, and half the argument for this model was throughput — it answered the original 2,800 questions in half the wall clock of its predecessor. That comparison is in the original post and this re-measurement does not touch it.

Setup

• NVIDIA GB10, 121 GiB unified memory, production services stopped for the duration
• vLLM vllm/vllm-openai:v0.27.1, Qwen/Qwen3.8-27B-FP8, --max-model-len 73728, --gpu-memory-utilization 0.55
• --speculative-config '{"method":"mtp","num_speculative_tokens":2}', draft acceptance 0.73–0.84 by subject
• lm-evaluation-harness, local-chat-completions, 8 concurrent requests
• mmlu_pro_fair: permissive answer extraction, truncating stop sequence removed
• No reasoning parser. Leaving it on costs 18 points on this model
• 5-shot CoT, greedy: lm-eval sends temperature: 0 and seed: 1234 on every request. 200 questions per subject, fourteen subjects