Four models on a DGX Spark: size predicted nothing

The speed comparison left the important question open. It measured which models are fastest on a DGX Spark and said, explicitly, that this does not tell you which one to run — because a model that is twice as fast and meaningfully worse is not a better model.

Here are the quality numbers for all four, at a sample size large enough to argue with.

The result

MMLU-Pro, 200 questions in each of fourteen subjects — 2,800 per model, 5-shot chain-of-thought, greedy decoding. Same machine, same harness, same answer-extraction rules, the same questions in the same order for every model. How the scoring works is written up separately; the short version is that every question ships its correct answer in the dataset and grading is a string comparison against it. No model judges another.

Every pairwise comparison, with confidence intervals on the difference:

That is an unusually clean shape. Every between-tier gap is significant and neither within-tier gap is. The four models are two pairs, and the pairs are six to eight points apart.

Size predicted nothing

The two tiers are 27B-and-35B on top, 30B-and-117B below. Parameter count across the four spans 27B to 117B and lands in no useful order: the smallest model is top of the table, the largest is bottom, and the two models that are statistically tied differ by 87 billion parameters.

Active parameters do not rescue it either. The 35B-A3B activates 3B per token and is in the top tier; gpt-oss activates 5B and is in the bottom one.

I keep having to relearn this. The original version of this post was written around the idea that a 117B model on a desk-side box ought to buy something, and the honest finding was that it did not. The fourth model makes the point sharper rather than softer — Muse Glimmer is a quarter of gpt-oss’s size and scores the same.

Where the tiers break

The aggregate hides more than usual, so here is every subject:

The tier split survives in twelve of fourteen subjects. The two that break it are worth more than the aggregate:

Engineering inverts completely. Both bottom-tier models beat both top-tier ones — gpt-oss at 63.5 and Muse Glimmer at 62.0 against the 27B’s 60.0 and the 35B’s 52.0. It is the hardest subject by average score, and the 35B’s worst result anywhere by a startling margin: 17.5 points below its own next-worst subject. If your work is engineering-shaped, this table’s aggregate ordering is actively misleading.

Math is a four-way wash, 87.5 to 90.5 across models six to eight points apart overall.

And the widest gap on the board is law, where the 27B scores 75.5 and Muse Glimmer scores 49.0 — a 26.5-point spread that the aggregate flattens into 7.6. Law is also the one subject harder than engineering for the bottom-tier pair; the tiers do not even agree on which subjects are difficult.

Per-subject figures are 200 questions each, so they carry roughly ±2 points of their own. Treat one-point differences within a row as nothing.

Now the speed

Single-stream throughput on a short chat turn, which is what interactive use actually feels like:

One caveat this table cannot show: the three original models were served on vLLM 0.19.0, and Muse Glimmer runs on 0.26.1rc1 because it needs a custom arm64 build to work at all. The quality comparison is unaffected — greedy decoding, and speculative decoding is lossless by construction since the full model verifies every drafted token — but the speed column is not a same-stack comparison, and I am not going to pretend it is.

gpt-oss-120b is no longer the thing to avoid

This is what the rewrite is about. Against Muse Glimmer, gpt-oss scores the same and runs 1.8× faster. On this hardware it strictly dominates — with one caveat I want stated here rather than buried: that tie is gpt-oss at its default medium reasoning effort against Muse Glimmer at its default high. Effort sweeps for both are running, and the last section says what that could change.

It still loses to the 35B-A3B on both axes — 6.5 points behind on quality and 26% slower — so the original finding survives as a statement about those two models. What did not survive is the leap from there to “the biggest model that fits is the worst one to run.” The biggest model here is mid-table on quality and second-fastest. The worst model to run is the 30B.

The two Qwens are still the same model, for these purposes

82.54% against 80.79%, with the interval on the difference running from −0.28 to +3.78. It does not exclude zero, but only just — and it is worth being straight about that, because at ten times the sample size this comparison went from “clearly nothing” to “probably something small.”

Three independent measurements now point the same way: an early run with strict extraction put them at 82.1% and 81.4%, the 280-question run at 81.8% and 79.3%, and this one at 82.54% and 80.79%. The 27B is ahead every time — by 0.7, 2.5 and 1.75 points — and no single run can call it.

So the practical question is not which is more accurate but what the 35B’s speed costs you in quality. The answer is 1.75 points measured, with an interval that neither excludes zero nor reaches four — and it runs 2.9× faster.

What I would run

Qwen3.6-35B-A3B. Fastest of the four by a wide margin, quality within two points of the best, and the quickest to cold-start. Its one real weakness is engineering, where it drops to 52% and finishes last — if that is your workload, this is the wrong recommendation and the table above says so.

Qwen3.6-27B if quality matters more than latency, or for law, history and philosophy, where it leads the bottom-tier pair by 18 to 26 points. It is 2.9× slower.

gpt-oss-120b for engineering, where it is the best of the four, or when you want a large-model behaviour profile MMLU-Pro does not capture. Not for general work — the 35B beats it on both axes.

Muse Glimmer 30B is hard to justify on these numbers alone. Same quality as gpt-oss at half the speed, and last place in law by a wide margin. Getting it running at all was the interesting part; the scores are not.

Total run time

Thirty-six hours of machine time for the quality half of this post, against about 26 minutes for the speed half.

That ratio is the reason this post exists separately from the speed comparison. Throughput is cheap to measure — a few hundred requests and you have a stable number. Quality at a sample size worth arguing with costs eighty times as much, which is why so many local-LLM comparisons report the first and assert the second.

The four models are not equally expensive. Qwen3.6-27B took 14.9 hours and gpt-oss-120b took 3.9 for the same 2,800 questions, because the first writes about four times as many tokens per answer.

Why the extraction rules needed fixing first

The first time I measured gpt-oss it scored 40%. That number was wrong, and the way it was wrong matters more than the result.

lm-eval’s MMLU-Pro extracts the final answer with a regex matching answer is (X). gpt-oss frequently ends with “the correct choice is (A)” or “The answer is (B).” instead. Correct answers, scored zero, for phrasing.

The fix is a permissive pattern applied identically to every model, taking the last match so a restatement at the end of a long chain-of-thought wins over anything said mid-reasoning. Before spending GPU time on it, I checked it against responses I had already logged:

That asymmetry is the whole reason to trust it. A looser grader that raised everyone’s score would just be a looser grader. This one recovers real answers from a model that phrases conclusions differently and leaves alone a model that already complies. There is a whole post about how bad the stock harness gets.

What this does not settle

MMLU-Pro is knowledge and reasoning, not coding. The 27B is documented as the stronger model on SWE-bench-shaped work, and any of these may have strengths this benchmark never touches — tool use, long-form generation, instruction following. A single benchmark ranks models on one dimension and should not be mistaken for a verdict.

The questions are public. Any of these models may have seen them in training, and there is no way to check from outside. This measures standing on a shared yardstick, not absolute capability.

Extraction failures are invisible here. A model that knows an answer but phrases it unrecognisably is scored wrong, and these runs did not log per-question output, so I cannot report how often that happened to each model. Later runs do log it. That is a gap in this post, not a solved problem.

Every model is at its own default thinking level, and the defaults are not the same. This is the caveat that most affects the headline. gpt-oss-120b ships a chat template defaulting to reasoning_effort: medium; Muse Glimmer’s defaults to reasoning_strength: high. So the two models I just called tied were measured at different points on their own effort curves — one at its ceiling, one in the middle. Both knobs live in the chat template’s default rather than in an API parameter, which is exactly why it is easy to benchmark a model at a setting you never chose.

I am measuring both curves now: gpt-oss at low and high, Muse Glimmer at low and medium, everything else held identical. Whether the tie survives depends on results I do not have yet, and this post will be updated with them rather than quietly left standing.

Single-stream, one configuration each. Both Qwens run MTP speculative decoding, Muse Glimmer runs DFlash at depth 15, and gpt-oss runs none — its checkpoint contains no draft heads, so the option is unavailable rather than declined.

Setup

• NVIDIA GB10, 121 GB unified memory
• vLLM 0.19.0 (nvcr.io/nvidia/vllm:26.04-py3) for the three original models; vLLM 0.26.1rc1, custom arm64 build, for Muse Glimmer. 32k context throughout
• lm-evaluation-harness 0.4.12, local-chat-completions, 8 concurrent requests
• mmlu_pro_fair: stock MMLU-Pro with permissive answer extraction and the truncating stop-sequence removed, identical for all four models
• 5-shot CoT, greedy, --limit 200 per subject = 2,800 questions per model
• Confidence intervals are the unpaired normal approximation. The models answered identical questions, so a paired test would have more power and these intervals are conservative — but the per-question records needed for one were not kept on these runs

Revision history

Source data: 2f908ad0ef7606040e4d9fd899ee050fe7f2e6223a2d9555