Local LLM Labs

Nemotron 3.5 at a 65,536-token cap: +7.6 points

dgx-spark Nemotron-3.5-Lightning-30B-A3B

Every quality number this site published before today was measured with answers capped at 3,072 tokens. That is lm-evaluation-harness’s default and it is what we inherited without arguing with it.

For Nemotron 3.5 it was wrong by 7.6 points on average and 36 points on one subject. The model did not change. The amount of room it was given to finish a sentence did.

The curve, on one subject

Engineering is where a generation cap bites first, because engineering answers are the longest. Same model, same 200 questions, five budgets:

Subject (accuracy %) 3,0728,19216,38428,67265,536
engineering 45.0 66.5 72.0 78.5 81.0
MMLU-Pro engineering, 200 questions per cell, Nemotron 3.5 against itself at five generation caps. Rendered from data/subject-sweep/.
Cap Accuracy Gain Tokens per answer Minutes Answers cut off
3,072 45.0 — 2,413 35 many
8,192 66.5 +21.5 4,362 64 many
16,384 72.0 +5.5 6,239 93 some
28,672 78.5 +6.5 7,816 114 7.0%
65,536 81.0 +2.5 8,559 124 0%

65,536 is where it stops, and that is measured rather than assumed. At that budget, all 200 answers ended because the model finished — finished_reason was stop on every one, and length on none. At 28,672 it was still cutting off 7% of them. When no answer is being truncated, a larger cap cannot change a score; it can only cost time. That is why this campaign does not go to 320k.

Three signals agree on saturation: truncation reaches zero, token growth flattens to +9.5% over the last step, and the accuracy gain collapses to +2.5.

All fourteen subjects

Subject (accuracy %) 3,072-token cap65,536-token cap Spread
engineering 45.0 81.0 36.0
chemistry 74.0 88.0 14.0
physics 78.0 89.0 11.0
law 48.5 59.0 10.5
business 79.5 88.5 9.0
biology 83.0 90.0 7.0
math 87.0 93.0 6.0
computer science 79.0 84.5 5.5
economics 80.0 85.0 5.0
psychology 76.5 79.0 2.5
health 73.5 71.5 2.0
other 67.0 68.5 1.5
philosophy 70.5 71.5 1.0
history 68.0 68.0 0.0
Mean 72.1 79.8 7.6
MMLU-Pro accuracy by subject, 200 questions per cell, Nemotron 3.5 at two generation caps. Spread is the gap between the two columns; on the Mean row it is the gap between the two column means, which is the mean delta. Sorted by spread, so the subjects the cap distorted most are at the top.

Twelve subjects up, one flat, one down. The mean goes from 72.11 to 79.75.

Engineering (+36.0), chemistry (+14.0), physics (+11.0) and law (+10.5) are where the damage was. Health fell 2.0 and history did not move.

That pattern is the finding, not the individual numbers. A cap does not penalise every subject equally — it penalises the subjects whose answers run long, and leaves the rest alone. Anyone comparing models on a benchmark with a default cap is comparing them on how concise they are as much as on what they know.

What it cost

Subject (tokens/answer) 3,072-token cap65,536-token cap
biology 1497 2134
business 1587 2353
chemistry 1859 3699
computer science 1562 2649
economics 1365 1612
engineering 2413 8559
health 1449 1751
history 1577 1860
law 2181 3464
math 1440 2796
other 1258 1914
philosophy 1513 1927
physics 1739 3369
psychology 1293 1395
Mean generated tokens per question at each cap. This is the budget the accuracy above was bought with.

Tokens per answer went from 1,624 to 2,820, and the full fourteen-subject sweep went from 5.6 hours to 10.0 hours on one DGX Spark at eight-way concurrency. Nearly twice the machine time for 7.6 points.

Every subject that gained also generated substantially more. The two that did not gain — health and history — grew least.

Total run time

Arm What ran Questions Tokens generated Machine time
3,072-token cap 14 subjects · 3,072-token cap 2,800 4,546,508 5.6 h
65,536-token cap 14 subjects · 65,536-token cap 2,800 7,896,599 10.0 h
Total 5,600 12,443,107 15.6 h
Machine time for both arms of this post, summed from data/subject-sweep/. Scoring time only: it excludes model loading, the rest period between subjects, and the 65,536 arm's timing probes at 8,192, 16,384 and 28,672.

Ten hours for the second arm, against 5.6 for the first. The work is the same on paper — 14 subjects × 200 questions = 2,800 questions either way — and the machine is the same DGX Spark running eight requests at a time.

What changed is how much the model wrote. At 3,072 it produced 4.5 million tokens across the sweep; at 65,536 it produced 7.9 million. Nearly twice the time buys almost entirely the same questions answered at greater length, and generation is the part that costs: at roughly 230 tok/s aggregate, six million extra tokens is about seven hours on its own.

The cost is not evenly spread. Engineering alone took 124 minutes of the ten hours — one subject consuming a fifth of the sweep, because its answers run to 8,559 tokens each against psychology’s 1,395. The subjects that gained the most are the subjects that cost the most, which is the same fact stated twice.

This is also why the campaign is scoped rather than universal. Re-measuring all six models on the fleet at this cap is an estimated 154 hours of machine time, during which the Spark serves nothing else.

The control

A result like this needs something to rule out “we changed the serving config and everything moved.” So the same treatment ran on gpt-oss-120b, a model whose answers are short enough that the 3,072 cap never bound them.

It moved 0.29 points, with its fourteen deltas split six up, six down and two flat, and its tokens per answer actually fell slightly (524 → 513).

That is what no effect looks like, and it is measured on the same machine in the same week with the same harness change. A model that cannot be cap-sensitive read as insensitive; a model that can be moved by eight points. The difference between those two results is the generation cap and nothing else.

What this changes

Nemotron’s published quality numbers on this site are wrong, and they are wrong in a direction that changed a ranking. At the 3,072 cap gpt-oss-120b (74.25) scored above Nemotron (72.11). At 65,536 Nemotron (79.75) scores above gpt-oss (74.54). The leaderboard had them the wrong way round.

The 3,072 records are not deleted. They stay in data/subject-sweep/ as the low-cap arm, which is what makes the size of the effect measurable rather than just embarrassing.

What this does not tell you

One model. Nemotron 3.5 is unusually fast, which is what makes a 65,536-token budget affordable to measure at all. A model at 20 tok/s would need days per subject. Whether other models gain this much is being measured; two of six are done.

200 questions per cell gives roughly ±9 points on a single subject difference. Engineering’s +36.0 and chemistry’s +14.0 clear that comfortably. Philosophy’s +1.0 and health’s −2.0 do not, and should be read as noise.

Not a claim about the model’s ceiling. 81.0 on engineering is what this model scores when it is allowed to finish, with this prompt, this harness and greedy decoding. It is not the best score obtainable from it.

The cap is fleet-specific. 65,536 is where Nemotron stops being truncated on these subjects. A model that reasons longer would need more, and a terser one is done far earlier — gpt-oss never needed more than the original 3,072.

Setup