Muse Glimmer at a 65,536-token cap: the model is its own control

Every quality number this site published before this campaign was measured with answers capped at 3,072 tokens. For Nemotron 3.5 that cost 7.6 points; for gpt-oss-120b it cost nothing at all, because that model never approaches the cap.

Muse Glimmer 30B is the fifth model through the same treatment, and it is the most useful of the five — not because of what it scored, but because of how its subjects divide. This model contains both the treatment and the control.

The result

74.93 to 77.07, +2.14. Ten subjects up, three flat, one down — the smallest mean gain of any cap-sensitive model in this campaign.

That number is close to useless on its own, and the reason is the next table.

The model is its own control

Sort the same fourteen subjects by how much longer the answers got, and the grid falls into two groups that barely overlap:

The six that grew are business, chemistry, computer science, engineering, math and physics. The eight that did not are biology, economics, health, history, law, other, philosophy and psychology.

This is the argument the whole campaign rests on, and here it is inside a single model. gpt-oss-120b was run as a separate control to rule out “we changed the serving setup and everything moved” — a good control, but a different model on a different day. Muse Glimmer needs no such comparison. Eight of its own subjects had their answer budget raised twenty-one-fold, used none of it, and scored the same. Whatever moved the other six, it was not the re-measurement itself.

It also sharpens what a generation cap actually does. A cap is not a difficulty setting or a quality knob. It truncates answers that would have run longer, and it is inert everywhere else. Two subjects in this grid demonstrate both halves:

• Engineering wrote 1,806 tokens per answer at the low cap and 3,730 at the high one, and went 62.0 → 75.5.
• History wrote 69 tokens per answer at the low cap and 68 at the high one, and went 66.5 → 66.0.

Sixty-eight tokens is not a broken run

History answering in 69 tokens looks like a model that has failed — a truncated response, a parser catching a fragment, a server returning nothing. It is none of those. The answers are one coherent sentence:

We refer to Wikipedia articles on ancient history for help. The most obvious material symbols of ancient state societies are monumental works. The answer is (D).

That is the complete generation for a question the model got right. The 3,072 baseline recorded 69 tokens per answer on the same subject; this run recorded 68. A model that was being truncated would not reproduce its own token count to within one token when the ceiling moves by a factor of twenty-one.

This model’s verbosity is extraordinarily subject-dependent: 68 tokens on history against 3,730 on engineering, a 55× spread under one prompt, one temperature and one harness. That is a much wider range than any other model in this campaign, and it is why the split above is so clean. Where it reasons, it reasons at length and the cap bit; where it does not, the cap was never in contact with anything.

The curve, on the one subject that matters

Engineering is where the cap did its damage, and it was measured at four budgets:

Engineering climbs 62.0 → 70.0 → 74.5 → 75.5, still gaining at the top of the range. That is unlike the two Qwen models, which were finished by 8,192 and scored no better after it. Muse Glimmer is a slow, verbose reasoner on this subject and it uses the budget it is given.

Law is the counterpart: 49.0 → 48.5 → 49.0 → 49.0, with tokens per answer sitting between 217 and 247 at every cap. Four measurements at four budgets, a one-point range, and no trend. Law is this model’s worst subject by a distance — 26.5 points behind Qwen3.6-27B — and the answer budget has nothing to do with it. Raising the cap does not fix a subject the model is simply bad at.

What this cost, and what that number does not mean

Here the post has to be careful, because the honest answer is that this comparison is confounded and the confound is mine.

The 3,072 baseline was served with the DFlash drafter this model ships with, drafting fifteen tokens ahead — the configuration the day-one post measured at 2.8–6.1× faster. The 65,536 arm was served without it. The campaign driver’s serving command for this cell was written without the --speculative-config its baseline had, and that was not caught until after the grid finished.

The numbers, stated so they cannot be mistaken for a finding:

The 9.8× increase in machine time is not the price of the higher cap. Answers got 1.78× longer, which is the cap’s doing; throughput fell 5.5×, which is the missing drafter. Multiply those and you get the 9.8×. A correctly served re-run would have cost something closer to eight hours.

Why the accuracy result survives this and the timing result does not. Speculative decoding is lossless by construction: the drafter proposes tokens, the full model verifies them, and anything it rejects is discarded. A drafter changes how fast tokens arrive, not which tokens they are. So the two arms are comparable on score and not comparable on time, and this post reports the score.

It is worth saying that this is a weaker position than the other four posts in the campaign, which compared arms that differed only in the cap. The driver has been fixed and now records the command it actually ran rather than the one a service catalogue thinks it would have run — which is what hid the difference in the first place. The misconfigured grid is kept rather than deleted: it is a valid measurement of accuracy and an invalid one of speed, and deleting it would hide both.

What this changes

Muse Glimmer’s published quality figure on this site is low by 2.1 points, 74.93 against 77.07. It does not change the model’s standing: it was fourth of four in the quality-versus-speed comparison and the models above it gained more, not less, from the same correction.

The day-one post’s two headline quality claims both hold. Law is still the weakness, and now with four measurements at four caps confirming the budget is not the cause. Computer science was one of its five worst subjects at the low cap and gains 4.5 points here — the one claim in that post the re-measurement softens, since some of that deficit was the cap rather than the model.

What this does not tell you

200 questions per subject, and the campaign has measured 8–10% of answers changing between identical runs on a different model. Treat single-subject moves under about 5 points as scatter: engineering’s +13.5 clears that, and nothing else in the grid does.

The two-group split is the finding, not the individual gains. No single subject in the “grew” group except engineering moved by more than the noise. It is the consistency of the pattern across six subjects, against eight that did not move at all, that carries the argument.

Nothing here is about speed, and given the drafter discrepancy this post is the wrong place to look for it. The speed measurements are in the day-one post, which measured the drafter properly.

One prompt. Five-shot chain-of-thought. A model this sensitive to subject in its verbosity may well be sensitive to prompt format too, and that has not been measured.

Setup

• NVIDIA GB10, 121 GiB unified memory, production services stopped for the duration
• vLLM vllm/vllm-openai:muse-glimmer-arm64-cu130, bf16, locally re-sharded weights, --max-model-len 73728
• The 65,536 arm ran without the DFlash drafter that the 3,072 arm used. See the cost section above; accuracy is comparable, timing is not
• lm-evaluation-harness, local-chat-completions, 8 concurrent requests
• mmlu_pro_fair: permissive answer extraction, truncating stop sequence removed
• 5-shot CoT, greedy: lm-eval sends temperature: 0 and seed: 1234 on every request. 200 questions per subject, fourteen subjects