Four models, fourteen MMLU-Pro subjects, one DGX Spark

Every model comparison on this site so far has ended with the same caveat: one aggregate number ranks models on one dimension and should not be mistaken for a verdict. This post is the attempt to stop hand-waving at that and measure it.

Four models, fourteen subjects, 200 questions in each — 11,200 questions and 35.8 hours of GPU time, with accuracy and throughput recorded for every one of the 56 cells. The three-model comparison that preceded this used 20 questions per subject, and three of its four per-subject findings did not survive being re-measured at ten times the sample.

The short version: the aggregate ranking is a decent guide in about ten subjects, misleading in three, and exactly backwards in one.

The aggregate, briefly

Two tiers, six to eight points apart, with every between-tier gap significant and neither within-tier gap significant. That much is covered elsewhere.

The column that does not appear in an aggregate table is the last one. Muse Glimmer and gpt-oss are statistically tied overall — and gpt-oss wins a subject outright while Muse Glimmer wins none. Two models with the same average are not interchangeable.

Every cell

Accuracy, 200 questions per cell. Bold is the best model in that row.

Sorted by spread, because spread turns out to be the most informative column in the table.

Which subjects can tell models apart at all

At the bottom of that table — math, chemistry, business — all four models land within three to four and a half points of each other. Given ±3 points of sampling error per cell, those subjects cannot distinguish these models. A benchmark suite made of them would report four equivalent systems.

At the top, law separates them by 26.5 points and history by 19.5.

The pattern is not subtle: the three widest spreads are law, history and philosophy, and the three narrowest are math, chemistry and business. Recall- and-judgement subjects discriminate; procedural ones do not. If you are choosing a model, the humanities rows carry nearly all the signal, even if your workload is technical — not because you need a model that knows case law, but because those are the rows where the models are actually different.

Engineering runs backwards

One subject inverts the aggregate completely:

Both bottom-tier models beat both top-tier ones. The 35B-A3B — second overall, and the model I have been recommending on this machine — finishes last by ten points, at 52.0. That is its worst subject by 17.5 points against its own next-worst, and the only place on this box where the Qwens are displaced at all.

Engineering is also the hardest subject in the set by mean score (59.4), with law second (61.6). Whatever it is measuring, it is not what the other thirteen subjects measure.

I do not have an explanation, and I want to be clear that I am reporting it rather than accounting for it. It reproduces across 200 questions with a ±3 point error bar, so it is not noise — but a single subject on a single benchmark is a lead, not a conclusion. If your work is engineering-shaped, the actionable version is: do not take the aggregate ranking on faith, and measure on your own tasks.

The inversion survives a second measurement; the tie-break does not. Re-run at reasoning_effort: high (see below), gpt-oss scores 62.0 on engineering rather than 63.5 — which ties Muse Glimmer exactly and gives up the outright win. Both bottom-tier models still beat both top-tier ones, by 2 and 10 points. So the finding that matters here holds at two different effort settings, while the “gpt-oss wins one subject” line in the aggregate table is specific to its shipped default and would become a shared win at high. That is a 1.5-point move inside a ±3-point band, so it is not evidence that effort hurts engineering — it is a reminder that ranking models by who came first in a row is fragile in a way that reporting the gap is not.

Now the speed

Throughput per subject, in tokens per second.

These are batched figures — lm-eval runs 8 concurrent requests — and are not comparable to the single-stream numbers elsewhere on this site. The 35B-A3B reads 136 tok/s here and 50.2 tok/s single-stream; both are correct, they measure different things. Use these to compare subjects within a model, and the single-stream numbers to decide what interactive use feels like.

Three of the four models look flat. The 27B stays inside 100–112 across every subject, the 35B inside 132–140, gpt-oss inside 98–110. Muse Glimmer swings from 59.1 to 124.5 — a factor of 2.1.

Before drawing anything from that, the table needs a correction that I nearly published without.

The speed table is partly measuring answer length

The harness computes throughput as generated tokens divided by elapsed time — and elapsed time includes prefill, while generated tokens do not include the prompt. So a subject where the model writes short answers to long questions reports low throughput even if the model is decoding at its usual rate.

That is not a small effect here. Across all 56 cells, how many tokens a model writes predicts its measured throughput at r = +0.68 — a stronger relationship than anything about the subject matter itself.

The Qwens look flat partly because they are stable decoders and partly because they write consistent-length answers everywhere. Muse Glimmer swings because it does not: on history it writes 69 tokens per question against a 2,222-token prompt, so 97% of the work in that cell is prefill and the “generation throughput” figure is close to meaningless.

So the honest reading of the speed table is: compare rows within the Qwen columns freely, and treat Muse Glimmer’s history and law cells as artifacts rather than measurements. I am leaving the table in with this caveat rather than deleting it, because the underlying numbers are real and the confound is worth showing.

Speculative decoding really is subject-dependent

The claim survives, but it needs evidence that is not the throughput column. Draft acceptance is recorded directly and is not affected by prefill at all:

Acceptance ranges from 0.199 on law to 0.316 on business — a 1.6× spread, measured directly, with no prefill in the denominator. That is the mechanism behaving as designed: speculative decoding buys speed by having a small drafter guess tokens the full model then verifies, so its payoff is a direct function of how predictable the text is. Legal reasoning is less predictable than business prose, the drafter misses more, and the machine does more full weight sweeps per token.

Excluding history — the one cell the metric mangles — acceptance and throughput correlate at r = +0.88, which is what the mechanism predicts.

The consequence is worth stating plainly: a headline speculative-decoding speedup is an average over content, and your content may not be average. The 23.3 tok/s in the day-one post is real, and on law-shaped text the drafter is landing a third fewer guesses.

There is a weaker second pattern: acceptance also tracks accuracy, at r = +0.39. That is not significant at fourteen points — the threshold is about 0.53 — so it is a hypothesis, not a finding. The appealing version is that a drafter fails to predict exactly the material the model handles poorly, which would make acceptance a cheap proxy for domain confidence. Appealing ideas of that shape need their own experiment, and this is not it.

How to use this

If you want one model and no further thought: the 27B. It wins nine of fourteen subjects and is never worse than third.

If you want speed and can accept about two points: the 35B-A3B, which is 2.9× faster single-stream and top-tier in thirteen subjects — with engineering as the one place it fails badly enough to matter.

If your workload is law-, history- or philosophy-shaped: the choice matters more than anywhere else in this table, and it is the 27B by 6 to 26 points.

If your workload is math, chemistry or business: pick on speed. These subjects cannot tell the four models apart, so the fastest adequate model wins by default.

If your workload is engineering-shaped: measure it yourself. This table’s aggregate ordering is inverted there, and I would not extrapolate from one subject on one benchmark to your actual tasks.

What this does not settle

Every model is at its own default thinking level, and the defaults differ. gpt-oss-120b ships reasoning_effort: medium; Muse Glimmer ships reasoning_strength: high. So the two models tied on aggregate were measured at different points on their own effort curves — one mid-range, one at its ceiling.

For gpt-oss that is now measured rather than assumed. The same 2,800 questions, same harness, same everything, with the chat template’s effort default changed from medium to high:

Turning the effort knob to maximum bought one extra token per question and finished slightly faster. The accuracy difference is a third of its own confidence interval, and no subject moved outside the ±3-point noise band — including math and physics, which were the ones I expected to move if anything did.

This makes the tie in the aggregate table more robust rather than less: gpt-oss at its maximum effort setting scores 74.86 against Muse Glimmer’s 74.93 at its default high, so the two models are indistinguishable at matched settings, not just at mismatched ones.

Muse Glimmer’s knob, measured the same way, does plenty — and it makes the gpt-oss result interpretable. Muse Glimmer ships reasoning_strength: high; here are all three settings, same 2,800 questions each:

Its default setting is the worst of the three on both axes at once. Medium scores +3.18 points above high and low scores +2.86, both outside the ±2.27 interval, while low finishes in 40% less wall clock on 44% fewer tokens. Medium and low are indistinguishable from each other (+0.32, well inside the band), so the practical reading is: anything below the default is better, and low is the cheapest way to get there.

For a model positioned for always-on local agents, that is worth acting on — one line in the chat template buys better answers for roughly half the tokens.

And it settles what gpt-oss’s flat result meant. Same harness, same benchmark, same greedy decoding, same five-shot prompting, same machine, same week:

No single explanation covers both. Five-shot anchoring and greedy decoding would have flattened Muse Glimmer’s knob too, and did not. The most likely reading is that gpt-oss’s reasoning_effort is not reaching the model in this vLLM build, with Muse Glimmer serving as the positive control that shows the harness detects the effect when it exists. A dedicated experiment crossing effort against shots and sampling is running to confirm that, and will get its own post.

One caveat I cannot close from this data: the default runs predate per-question logging, so I can measure answer-extraction failure on the medium and low runs (1.2% for Muse Glimmer) but have no figure for the high runs. If high’s longer answers trip the extraction regex more often, some of that 3-point gap is mechanical rather than a real difference in knowledge. That is a re-run, not a reinterpretation.

The questions are public. Any of these models may have seen them in training, and there is no way to check from outside. This measures standing on a shared yardstick, not absolute capability — which is why every model here answered identical questions in identical order rather than being compared via published model-card figures.

Extraction failures are invisible. A model that knows an answer but phrases it unrecognisably is scored wrong, and these runs did not log per-question output, so I cannot report how often that hit each model. Muse Glimmer’s 69 tokens per question on history — against 1,806 on engineering — is exactly the shape that would hide such a problem, and I cannot currently rule it out. Later runs log samples; these do not.

±3 points per cell. At 200 questions a subject carries roughly three points of sampling error at one standard error. Treat single-point differences within a row as nothing. The 26.5-point and 19.5-point spreads are real; the 3.0-point one is not.

The throughput metric has a known flaw, described above: it divides generated tokens by elapsed time that includes prefill, so short-answer subjects under-report. It is fine for comparing models with similar answer lengths and unreliable for Muse Glimmer’s shortest cells. A cleaner per-subject speed comparison would time decode separately from prefill, which this harness does not currently do — that is a fix for the next revision, not a reinterpretation of these numbers.

One machine, one configuration each. Both Qwens run MTP speculative decoding, Muse Glimmer runs DFlash at depth 15, gpt-oss runs none — its checkpoint contains no draft heads.

Total run time

Thirty-six hours for 11,200 questions, on one DGX Spark running eight requests at a time.

Where it goes is not where you would guess. gpt-oss-120b is the largest model here at 117B parameters and it swept fourteen subjects in 3.9 hours; Qwen3.6-27B is a quarter its size and took 14.9. The difference is not speed, it is verbosity — gpt-oss wrote 1.5 million tokens across the sweep and Qwen3.6-27B wrote 5.8 million. A benchmark’s cost tracks how much a model says, not how big it is.

That ratio is why the sample size is 200 questions per subject rather than the full 12,032-question set. Every model measured on that would be roughly sixty times these figures — Qwen3.6-27B alone would run for five weeks.

Setup

• NVIDIA GB10, 121 GB unified memory
• vLLM 0.19.0 (nvcr.io/nvidia/vllm:26.04-py3) for the Qwens and gpt-oss; vLLM 0.26.1rc1, custom arm64 build, for Muse Glimmer. 32k context throughout
• lm-evaluation-harness 0.4.12, local-chat-completions, 8 concurrent requests
• mmlu_pro_fair: stock MMLU-Pro with permissive answer extraction and the truncating stop-sequence removed, identical for all four models. Full scoring method
• 5-shot CoT, greedy, --limit 200 per subject = 2,800 questions per model
• Runs were paced with idle periods equal to half each subject’s runtime, so the machine was not held at load for the full 35.8 hours and other work could claim the GPU between subjects
• Result files: data/subject-sweep/, written after every subject