Local LLM Labs

gpt-oss-120b does not care about the answer budget

dgx-spark gpt-oss-120b

This is the boring half of a result, and it is the half that makes the other half mean anything.

Nemotron 3.5 gained 7.6 points when its answers were allowed to run to 65,536 tokens instead of 3,072. On its own that number proves less than it looks like it does. Raising a generation cap means restarting the server with a longer context, re-running the harness, and occupying the machine for twice as long — plenty of room for something other than the cap to be responsible.

So the same change ran on a model that cannot be cap-sensitive.

The measurement

gpt-oss-120b’s answers are short. At the original 3,072-token cap it averaged 524 tokens per answer, nowhere near the ceiling. If a generation cap is what moved Nemotron, this model should not move at all.

Subject (accuracy %) 3,072-token cap65,536-token cap Spread
other 64.5 69.0 4.5
history 60.0 64.0 4.0
law 52.5 49.0 3.5
business 81.5 84.0 2.5
math 89.5 87.5 2.0
physics 82.0 80.0 2.0
economics 83.5 82.0 1.5
philosophy 67.0 68.5 1.5
biology 83.5 84.5 1.0
computer science 82.5 83.0 0.5
engineering 63.5 63.0 0.5
psychology 77.5 77.0 0.5
chemistry 80.5 80.5 0.0
health 71.5 71.5 0.0
Mean 74.3 74.5 0.3
MMLU-Pro accuracy by subject, 200 questions per cell, gpt-oss-120b at two generation caps 21x apart. Spread is the unsigned gap between the two columns, so a subject that fell shows the same as one that rose by as much; the Mean row's spread is the gap between the column means and does carry the sign.

74.25 to 74.54. A mean change of +0.29 points from twenty-one times the answer budget.

Why this reads as nothing rather than as a tiny gain

Three things, and the first is the weakest:

The size. +0.29 on a mean of fourteen subjects is far inside what 200 questions per cell can resolve. That alone would only mean “we cannot see it.”

The direction. The fourteen deltas split six up, six down, two flat. A real effect pushes subjects the same way — that is what it did on Nemotron, where twelve of fourteen moved up together. Splitting evenly is what scatter looks like, and it is the same reasoning that settled the reboot question and the 1-bit quantisation question.

The mechanism. Tokens per answer went down, 524 to 513, and the sweep finished slightly faster (3.9 hours against 3.7). A model given twenty-one times more room used marginally less of it. There is no pathway by which the cap could have changed the score, because the cap was never reached.

Subject (tokens/answer) 3,072-token cap65,536-token cap
biology 386 428
business 617 565
chemistry 822 843
computer science 441 422
economics 310 291
engineering 1409 1413
health 283 270
history 333 293
law 560 546
math 586 592
other 306 284
philosophy 401 398
physics 654 617
psychology 226 217
Mean generated tokens per question at each cap. The budget rose 21x; what the model actually used did not move.

What it is for

A benchmark campaign that only measures where it expects to find something cannot tell a real effect from a change in conditions. This is the negative control, and it was run on the same machine, in the same week, with the same harness, the same serving change, and the same raised --max-model-len.

The pair is the result:

Mean @3,072 Mean @65,536 Change Direction of the 14 deltas
Nemotron 3.5 72.11 79.75 +7.64 12 up, 1 flat, 1 down
gpt-oss-120b 74.25 74.54 +0.29 6 up, 6 down, 2 flat

One model moved by eight points and one did not move at all, under an identical intervention. That difference is the generation cap.

It also inverts the ranking. At the low cap gpt-oss outscored Nemotron, 74.25 to 72.11. At the honest cap Nemotron outscores gpt-oss, 79.75 to 74.54. Neither model changed; the benchmark did.

Total run time

Arm What ran Questions Tokens generated Machine time
3,072-token cap 14 subjects · 3,072-token cap 2,800 1,466,513 3.9 h
65,536-token cap 14 subjects · 65,536-token cap 2,800 1,435,582 3.7 h
Total 5,600 2,902,095 7.6 h
Machine time for both arms, summed from data/subject-sweep/. Scoring time only: it excludes model loading and the rest periods between subjects.

The high-cap arm was faster. 3.7 hours against 3.9, for the same 2,800 questions on the same machine.

That is not a speedup and nothing was optimised — it is the clearest statement of the finding available. A generation cap costs time only when it is reached, and this model never reaches it. Tokens generated barely moved (1.47 million against 1.44 million), so the two arms did the same work and finished in about the same time, with the difference well inside normal run-to-run variation.

Set against Nemotron’s 5.6 to 10.0 hours under an identical change, the contrast is as sharp in machine time as it is in accuracy. One model was being cut off and had a great deal more to say; the other was already finishing.

The practical version

If your model is terse, the generation cap is not your problem. gpt-oss-120b scores the same at 3,072 as at 65,536, and every hour spent re-measuring it at the higher cap bought nothing. Its published numbers on this site were already right.

If your model reasons at length, the cap is measuring your model’s patience rather than its knowledge. The way to tell them apart is not to argue about it: check what fraction of answers ended because the model stopped, against what fraction ended because the harness did.

That check costs nothing and would have saved this site several weeks of wrong numbers.

What this does not tell you

A negative result at this sample size is not proof of zero. 200 questions per cell puts roughly ±9 points around a single subject comparison. A one-point real effect would be invisible here. The claim is that gpt-oss shows no cap sensitivity that this measurement can detect, and there is a mechanism — answers well short of the ceiling — explaining why it should not.

One model, one harness. gpt-oss-120b at MXFP4 on vLLM, five-shot CoT, greedy. A different prompt could make the same model verbose enough to be truncated.

Reasoning effort was left at its default. This model ignores that setting anyway, which is a separate measurement, but it does mean this post does not test a configuration where gpt-oss might generate substantially more.

Setup