Qwen3.6-27B at a 65,536-token cap: 85.96, the highest score on this site

Qwen3.6-27B is the dense model in a fleet that mostly is not. It was already the best scorer on this site at the old 3,072-token answer cap, and re-measuring it with that cap effectively removed puts it at 85.96 across fourteen MMLU-Pro subjects — the highest fourteen-subject mean recorded here, ahead of Nemotron 3.5’s 79.65 and gpt-oss-120b’s 74.54.

It is also, by a distance, the most expensive of the four to measure: 19 hours of scoring time for one grid.

Two things in this post are worth more than the headline. One is that the model was done gaining by 8,192 tokens, so 65,536 bought insurance rather than points. The other is that the run-to-run noise on this benchmark is large enough to eat its lead over the model in second place.

All fourteen subjects

Ten subjects up, two flat, two down. Engineering gained 19.0, chemistry 8.5, physics 7.0 and business 6.0. Law fell 2.0 and economics 0.5.

Physics at 95.5 is the highest single-subject cell any model has recorded on this site, against 93.5 for the second-placed Qwen3.6-35B-A3B on the same subject and the same questions.

Unlike the 35B-A3B, this model’s gain is not one subject in disguise. Engineering’s +19.0 is worth 1.36 of the 3.42-point mean gain; drop engineering and the mean still moves 84.27 to 86.50. Chemistry, physics and business each contribute as much as engineering does after the division.

Where the cap actually stopped mattering

The campaign chose 65,536 because that is where Nemotron 3.5 stopped being truncated. This model needed a small fraction of it:

60.0, then 81.5, then 82.5, then 79.0. The first step is worth 21.5 points. Everything after it is worth nothing — and the three high-cap readings spread across 3.5 points in no particular order, with the largest budget scoring the lowest of the three.

That last detail is the useful one. Three measurements of the same 200 questions at caps that no longer bind produced 81.5, 82.5 and 79.0. Whatever separates those numbers, it is not the answer budget, because the model is not using the extra budget:

2,770 tokens at 3,072; 4,446 at 8,192; then 4,645 and 4,543. The model writes what it writes. After 8,192 the cap is not part of the experiment any more, and the differences between those arms are the benchmark’s own scatter.

This post reports no truncation percentage, and cannot: the sample files this site records do not carry a finish reason, so the fraction of answers cut off by the harness is not recoverable from them. Nemotron’s 7.0%-to-0% figures came from a dedicated probe that was never repeated for this model. Flat token counts and flat scores are consistent with saturation without proving it.

How much noise is in a subject score

Law fell 2.0 points here and 5.0 on the 35B-A3B, while nearly everything else rose. Because a bigger answer budget cannot cost a model points on the merits, law was re-run on both models: same 200 questions, same cap, same serving command, recorded as a separate arm.

Five readings: 75.5, 74.5, 72.5, 73.5, 74.5. The paired comparison between the two 65,536 runs is the direct measurement — of 200 questions, 184 answers agreed and 16 flipped, 9 from wrong to right and 7 from right to wrong, an exact McNemar p of 0.80.

About 8% of this model’s law answers are unstable between identical runs. The 35B-A3B was worse at 10%. Neither model lost anything real on law.

Both runs decoded greedily — lm-evaluation-harness sends temperature: 0 and seed: 1234 on every request — so none of this is sampling. It is worse than sampling, in a way that is easy to miss: of the 200 answers, not one was byte-identical between the two runs. They diverge after a median of 366 characters of an answer averaging 10,844 characters. Under continuous batching the composition of a batch depends on timing, the order of floating-point reductions depends on the batch, and an argmax that is nearly tied occasionally falls the other way. Over a long answer that compounds into a different chain of reasoning and, about 8% of the time, a different letter.

This is the number to carry into every other table on this site. A subject score from a single run has a few points of slop in it, which means:

• Subject-level deltas under about 5 points are not findings. In the table above, engineering, chemistry, physics and business clear that. Math’s +2.5, psychology’s +2.0 and computer science’s +2.0 do not.
• A 1.64-point gap between two models’ means is not a ranking. This model’s 85.96 against the 35B-A3B’s 84.32 is inside the noise, as was the 82.54 against 80.79 at the low cap. The two are tied as far as this benchmark can tell, which is what the MoE-versus-dense post concluded from the low-cap arm and remains true at the high one.

The honest version of the headline is therefore: 85.96 is the highest number recorded here, and it is not a demonstration that this is the best model on the fleet.

What it cost

14.9 hours to 19.0 hours, at 2,074 to 2,466 generated tokens per answer. In wall-clock terms the second arm occupied the machine for 21.2 hours once model loading and the rest periods between subjects are counted.

Part of that is not extra tokens. Aggregate throughput fell from 108 tok/s to 101 tok/s between the arms — the same model on the same machine, generating longer answers at eight-way concurrency, where a longer answer means a larger KV cache per slot. The dense 27B is the slowest model in the campaign and the most expensive to benchmark: 19 hours against Nemotron’s 10 and gpt-oss-120b’s 3.7 for the identical 2,800 questions.

That cost is the practical argument against re-measuring everything at 65,536 rather than at a cap this model demonstrably reaches.

What this changes

This model’s published quality figures on this site were low by 3.4 points on the mean and 19 on engineering. The 3,072 records stay in data/subject-sweep/ as the low-cap arm; they are the evidence for the size of the effect, not an embarrassment to be deleted.

It does not change any ranking. Qwen3.6-27B was first at 3,072 and is first at 65,536, and both of its leads are too small to be called leads.

What this does not tell you

Nothing here is about speed. This is the slowest model in the campaign, and a quality table that ignores that is half an answer. The throughput comparison lives in the MoE-versus-dense post.

200 questions per subject, and a measured 8% answer instability on the one subject that was repeated. Treat single-subject differences under 5 points as scatter.

The 8,192 finding is this model on these subjects, not a recommendation for every model. Nemotron 3.5 was still gaining 6.5 points between 16,384 and 28,672. The transferable claim is that the right cap is model-specific, and cheap to find by checking whether answers are still growing.

One serving configuration. vLLM with FP8 weights and MTP speculative decoding at two draft tokens, --max-model-len 73728, eight concurrent requests. Decoding is greedy and seeded, and it still does not reproduce — see the law section. A run at one-way concurrency, or without speculative decoding, might be steadier; that has not been measured.

Setup

• NVIDIA GB10, 121 GiB unified memory, production services stopped for the duration
• vLLM (nvcr.io/nvidia/vllm:26.04-py3), Qwen/Qwen3.6-27B-FP8, --max-model-len 73728 (65,536 output plus a ~2,279-token five-shot prompt)
• --speculative-config '{"method":"mtp","num_speculative_tokens":2}'
• lm-evaluation-harness, local-chat-completions, 8 concurrent requests
• mmlu_pro_fair: permissive answer extraction, truncating stop sequence removed
• 5-shot CoT, greedy: lm-evaluation-harness sends temperature: 0 and seed: 1234 on every request. 200 questions per subject, fourteen subjects
• The law re-run is recorded as a separate arm (maxgen-65536-law-rerun) and is excluded from the fourteen-subject figures above