MMLU-Pro engineering mostly measures truncation

Qwen3.6-35B-A3B scores 52.0 on MMLU-Pro engineering. It also scores 83.0. Same model, same weights, same machine, same 200 questions, same greedy decoding. The only thing that changed was max_gen_toks — how many tokens lm-eval lets the model produce before cutting it off — which went from the 3072 this site had been using to 8192.

Thirty-one points. That is larger than the gap between any two models in our four-model comparison, and it came from a setting most people never write down.

This post is the sweep that came out of that: six models, two subjects, three caps. Eighteen cells, about 45 hours of GPU time.

Why engineering and law

Not every subject can be truncated, because not every subject provokes a long answer. The two chosen here sit at opposite ends of that:

• Engineering produces the longest answers on this fleet. Multi-step derivations, unit conversions, models talking themselves through a calculation.
• Law produces some of the shortest. The reasoning is retrieval-shaped — either you know which doctrine applies or you do not.

If truncation is what a raised cap fixes, engineering should move and law should not. That is the prediction the sweep was built to test, and it is a falsifiable one: had both subjects moved together, the effect would have been something else — a decoding change, a prompt-length artefact — and not truncation at all.

The three that clear the noise

Three models, +31.0, +21.5 and +21.5 on engineering going from 3072 to 8192. At 200 questions the 95% interval on a difference like that is about ±9 points, so all three are comfortably real.

Nemotron’s is the one worth dwelling on. Its 45.0 was the worst engineering score we had ever published, and the post that reported it treated it as a genuine weakness of the model. It was not. Give the model room to finish its reasoning and it scores 66.5, which is mid-pack. We were measuring our own harness and attributing the result to NVIDIA.

The three that do not

Muse Glimmer gains 8.0 on engineering, gpt-oss 5.0, Qwen3.8 3.5. None of those clears the ±9-point interval at this sample size. Read individually, all three are “no measurable effect”.

Read together, they are not nothing. All six engineering deltas point the same way, and six of six in one direction happens 3% of the time by chance. The weak results are consistent with a real effect too small for 200 questions to resolve — which is a different statement from “no effect”, and the honest version is that we cannot separate the two without more questions per cell.

The token counts say which is which

Accuracy is the noisy measurement. How many tokens a model actually generated is not, and it answers a cleaner question: did raising the ceiling change the model’s behaviour at all?

Engineering, mean generated tokens per question:

Five of six models immediately spent the extra budget. gpt-oss did not — seven tokens’ difference across 200 questions, and its wall-clock was 43 minutes at every one of the three caps. It was never hitting the 3072 ceiling in the first place, so raising it could not possibly have helped, and its +5.0 on engineering has no mechanism behind it. That one really is noise.

This also gives the other five a floor: whatever their accuracy did, the cap was demonstrably binding on them at 3072. For Muse Glimmer and Qwen3.8 the extra tokens simply did not convert into extra correct answers at a rate we can detect.

Law behaved as predicted

Six law deltas from 3072 to 8192: +1.0, −0.5, −0.5, +7.0, −1.0, +9.0. Three up, three down, and only two of them larger than a couple of points. That is what no effect looks like, against the same models in the same week.

The prediction held. A cap only matters when answers run into it, and law answers do not.

There is a hint of something in the other direction at the top end — going from 8192 to 16384, law falls for Qwen3.6-27B (74.5 → 72.5), Qwen3.6-35B (69.0 → 68.0) and gpt-oss (53.5 → 52.5). Every one of those is well inside the noise and we are not claiming it. It is worth a properly powered look later, because “more room to reason makes retrieval-shaped answers worse” would be an interesting result if it survived one.

What it costs

The cap is not free, and on two models it is startlingly expensive.

Engineering, one 200-question category:

Four models hold their throughput within a few percent across all three caps. Two collapse: Qwen3.8 halves, Muse Glimmer loses three quarters of its speed.

We do not know why. It is not contention — four other models measured on the same machine in the same window held steady, which is about as good a control as this setup offers. It is not a serving-config change either; we checked, and Muse Glimmer’s docker command is byte-identical across all three of its runs. Something about how those two models behave when generating long sequences costs them dearly, and identifying it is a separate piece of work.

The practical consequence is blunt. Muse Glimmer’s engineering category went from 49 minutes to 310 minutes for a gain we cannot even confirm is real.

What to take from this

If you publish an MMLU-Pro number, publish the generation cap next to it. Without it the score is uninterpretable. A 31-point swing from a harness setting is larger than most of the model-to-model differences people write posts about, and lm-eval’s own default is low enough to truncate five of the six models we tested.

A low engineering score is a truncation suspect first. Before concluding a model is weak at multi-step reasoning, check whether it was allowed to finish. We got this wrong in public and had to come back and correct it.

Updated 2026-08-31: the recommendation below was wrong, and this post is where it was published. It said 8192 was enough because every gain that cleared the noise arrived by then. It does not. Extending the same measurement on Nemotron 3.5 to a 65,536-token cap took MMLU-Pro engineering from 66.5 at 8192 to 81.0 — another 14.5 points past the figure this post called a default, with the largest single step at 16384, exactly where it claimed “another point or two at best.”

The error was reading a curve that was still climbing as one that had flattened. Three caps is not enough to see saturation, and the two Qwen3.6 models that did flatten by 8192 were the ones I generalised from.

The cap where truncation actually stops is 65,536: at that budget zero of 200 engineering answers ended because they hit the ceiling, against 7.0% at 28,672. A separate post covers the full curve and what it did to all fourteen subjects. The paragraph below is left as published rather than rewritten, because a benchmark site that quietly edits its recommendations is worth less than one that shows them being corrected.

8192 looks like the right default for this fleet, and we have switched to it. Every gain that clears the noise arrives by 8192; going on to 16384 buys another point or two at best and costs real hours.

Total run time

About fifty hours of machine time to move two subjects across two caps.

The cost is not distributed anything like the models are. gpt-oss-120b did both its arms in two hours combined; Qwen3.8-27B took eighteen. Same subjects, same question count, same machine — the difference is entirely how much each model writes when the ceiling is lifted, which is the finding this post is about, arriving as a bill.

That is also why this post covers two subjects rather than fourteen. At the observed rate the full grid would be roughly seven times these hours, or about two weeks of the machine doing nothing else.

What this does not tell you

Two subjects, not fourteen. They were picked as opposite extremes on answer length. The other twelve sit somewhere in between and were not measured at these caps.

200 questions per cell gives roughly ±9 points on a difference, which is why half this post is about what we cannot resolve. Distinguishing “small real effect” from “no effect” for Muse Glimmer and Qwen3.8 needs several times the sample, which is several times 45 hours.

Nemotron and Qwen3.8’s 3072 baselines predate serving-command recording, so we cannot verify byte-for-byte that they were served identically to their cap runs. The other four models are verified identical. The three significant results include Nemotron, so treat that one as slightly softer than the two Qwen3.6 results.

Throughput figures here are advisory. These runs were measured without the broker’s quiescence check, alongside production services on the same machine. Accuracy is unaffected by a busy neighbour; tokens per second is not. The accuracy column is the claim; the speed column is context.