Qwen3.6-35B-A3B at a 65,536-token cap: +3.5 points, and 8,192 would have done it

Every quality number this site published before last week was measured with answers capped at 3,072 tokens — lm-evaluation-harness’s default, inherited without argument. On Nemotron 3.5 that cap cost 7.6 points, and 36 on one subject.

This is the same treatment applied to Qwen3.6-35B-A3B: fourteen subjects, 200 questions each, the answer budget raised twenty-one fold. The mean moves 80.79 to 84.32.

The headline is the least interesting part. The useful findings are where those points came from, and where they had already stopped coming.

All fourteen subjects

Nine subjects up, one flat, four down. Engineering gained 30.5, chemistry 8.5, physics and math 6.0 each. Law lost 5.0 and other lost 2.0 — both of which turn out to be noise, and one of which is measured below.

The gain is one subject wearing a mean

Engineering’s +30.5, spread over fourteen subjects, is +2.18 of the +3.53. Drop engineering entirely and the mean moves 83.00 to 84.46 — a gain of 1.46 points across the other thirteen subjects, of which chemistry, physics and math supply nearly all.

This is the shape the cap campaign keeps producing. A generation cap does not tax subjects evenly. It taxes the subjects whose answers run long, and leaves the rest alone: engineering answers here average 5,031 tokens against psychology’s 1,913. A benchmark run under a default cap is partly measuring how concise a model is, and that penalty lands almost entirely on one column of the table.

The curve stops well before the cap does

The campaign settled on 65,536 because that is where Nemotron 3.5 stopped being truncated. For this model that budget was over-provisioned by a factor of eight, and the intermediate arms measured during the earlier engineering investigation show it:

52.0, then 83.0, then 84.5, then 82.5. The entire effect lands in the first step. Going from 3,072 to 8,192 is worth 31.0 points; the eightfold increase from 8,192 to 65,536 is worth −0.5, which is to say nothing, in the direction that makes the point.

The token counts say the same thing more directly:

Answers grow 2,790 to 4,659 across the first step — a 67% jump, which is the model finishing sentences it was previously cut off mid-way through. After that they flatten: 4,955 at 16,384 and 5,031 at 65,536, a further 8% for an eightfold increase in room. The model is not being truncated at 8,192 in any way that matters to its score.

One caveat on that claim, stated because the Nemotron post could make a stronger version of it: there is no truncation percentage in this post. The sample files this site records do not carry a finish reason, so “what fraction of answers ended because the harness stopped them” is not recoverable from the data here — Nemotron’s 7.0% and 0% came from a dedicated probe that was not repeated for this model. What is measured is the token growth flattening and the score not moving. Those are consistent with saturation; they do not prove zero truncation.

The law scare

Law fell 5.0 points, and every other subject in the table was either flat or up. A larger answer budget cannot cost a model points on the merits, so that number was either noise or something specific about long answers in that subject. The campaign could not tell which, so law was re-run: same 200 questions, same cap, same serving command, recorded as its own arm so it could not be quietly merged into the grid.

67.5 on the re-run, against 64.5 the first time. Because both runs answered the identical questions, the totals are not the useful comparison — the paired one is. Of 200 questions, 180 answers agreed and 20 flipped, 13 from wrong to right and 7 from right to wrong. An exact McNemar test puts that split at p = 0.26: what a coin toss looks like.

So roughly one law answer in ten is unstable between runs, and the −5.0 was the low end of a noisy pair. Nothing about the cap hurt this subject.

Both runs decoded greedily — lm-evaluation-harness sends temperature: 0 and seed: 1234 on every request — so none of this is sampling. It is worse than sampling, in a way that is easy to miss: of the 200 answers, not one was byte-identical between the two runs. They diverge after a median of 1,018 characters of an answer averaging 11,693 characters. Under continuous batching the composition of a batch depends on timing, the order of floating-point reductions depends on the batch, and an argmax that is nearly tied occasionally falls the other way. Over a long answer that compounds into a different chain of reasoning and, a tenth of the time, a different letter.

The number worth taking away is not law’s score. It is the instability itself. A single-run subject score at this cap carries a few points of slop, which means subject-level deltas smaller than about 5 points are not findings, and model-to-model gaps of 1–2 points on a mean are not rankings. That applies to this post’s own table: chemistry’s +8.5 clears it, philosophy’s +1.0 does not.

other’s −2.0 was not re-run and is left as it is: below the resolution of this measurement, and reported rather than explained away.

What it cost

12.5 hours to 15.5 hours for the same 2,800 questions, at 2,207 to 2,653 generated tokens per answer. Throughput was not the variable — the model ran at 137 tok/s in the first arm and 133 in the second, on the same machine at eight-way concurrency. The extra three hours is extra tokens, nothing else.

Engineering alone accounts for 2.0 of those hours, against 1.1 at the low cap. The subject that gained the most is the subject that cost the most, which is the same fact said twice.

What this changes

This model’s published quality numbers on this site are low, by 3.5 points on the mean and by 30 on engineering. The 3,072 records stay in data/subject-sweep/ as the low-cap arm — deleting them would remove the evidence for the size of the effect.

The ranking against Qwen3.6-27B does not change: the dense 27B scored above this model at 3,072 (82.54 to 80.79) and scores above it at 65,536 (85.96 to 84.32). Both gaps are larger than the gap between the two caps’ means but, at 1.64 points on the second pair, inside the run-to-run noise this post measured on law. These two models are not separated by these measurements, and the MoE-versus-dense post that called quality a tie at the low cap is still the honest reading at the high one.

What this does not tell you

Nothing here is about speed. This model’s case against the 27B was always throughput, and that comparison is unchanged and lives in the MoE post.

200 questions per subject resolves roughly ±9 points on a single-subject comparison, and the law re-run showed the practical figure is not much better than that. Engineering’s +30.5 and chemistry’s +8.5 clear it. The four subjects that “fell” do not.

The 8,192 finding is this model on these subjects. It is not a recommendation to run every model at 8,192 — Nemotron 3.5 was still gaining 6.5 points between 16,384 and 28,672. The general claim is that the right cap is model-specific and cheap to find, not that 8,192 is the answer.

One serving configuration. vLLM with MTP speculative decoding at two draft tokens, --max-model-len 73728, eight concurrent requests. Decoding is greedy and seeded, and it still does not reproduce — see the law section. A run at one-way concurrency, or without speculative decoding, might be steadier; that has not been measured.

Setup

• NVIDIA GB10, 121 GiB unified memory, production services stopped for the duration
• vLLM (nvcr.io/nvidia/vllm:26.04-py3), --max-model-len 73728 (65,536 output plus a ~2,279-token five-shot prompt)
• --speculative-config '{"method":"mtp","num_speculative_tokens":2}', draft acceptance 0.79–0.90 by subject
• lm-evaluation-harness, local-chat-completions, 8 concurrent requests
• mmlu_pro_fair: permissive answer extraction, truncating stop sequence removed
• 5-shot CoT, greedy: lm-evaluation-harness sends temperature: 0 and seed: 1234 on every request. 200 questions per subject, fourteen subjects
• The law re-run is recorded as a separate arm (maxgen-65536-law-rerun) and is excluded from the fourteen-subject figures above