Model comparison

Every number on this page was measured on hardware we own, and traces to a machine-generated run file in the repository — nothing here is a vendor claim, a reported figure, or an estimate. The tables are generated from those files at build time, so the page updates whenever a benchmark lands. Click any column header to sort.

• Fastest Nemotron-3.5-Lightning-30B-A3B 142.4 tok/s short chat, single stream
• Fastest long context Nemotron-3.5-Lightning-30B-A3B 87.6 tok/s long-context prompt
• Highest quality Qwen3.6-27B 82.5% MMLU-Pro, 14 subjects
• Leanest answers gpt-oss-120b 524 tok tokens per answer

All figures below were measured on NVIDIA DGX Spark (GB10).

Throughput

End-to-end tokens per second, best observed result per configuration. Higher is better.

Short chat is a 128-token prompt with 256 out — what interactive use feels like. Code gen is 512 in, 1024 out. Long ctx is an 8,192-token prompt, which is prefill-bound and the number most reviews never publish. A dash means that scenario has not been run for that configuration, not that it failed.

Quality

MMLU-Pro, all fourteen subjects, 200 questions each, at every model's default configuration.

Read the accuracy column with a caveat. These sweeps ran at a 3,072-token generation cap, and we later measured that the cap itself truncates long answers badly — engineering scores moved by up to 31 points when it was raised to 8,192. See the generation-cap post. The ranking above is consistent within itself, because every model ran under the same cap, but the absolute numbers understate every model and understate the verbose ones most. Re-running the full grid at 8,192 is pending.

Sample size is 200 questions per subject, roughly ±6 points at 95% on a single subject and tighter on the fourteen-subject mean. Gaps of a point or two are noise.

Some models appear in the speed table but not here. Qwen3.8-Flash-Next and DeepSeek-V4-Flash have not been swept across all fourteen subjects — they were measured on engineering and law only, at a higher generation cap. Their accuracy is in the table below rather than missing.

Measured, but not in the ranking

These runs are real measurements that cannot be lined up against the table above — a different inference engine, a different quantisation family, or a deliberately contended machine. Putting them in the same column would produce a tidy table that misleads, so they sit here with their configuration stated.

The same two subjects at a workable generation cap

The table above runs at a 3,072-token cap. These ran at 8,192, on engineering and law only — the subject where truncation bites hardest and the one where it does not. Same subjects, same cap, so these rows compare with each other.

The mean here is of two subjects, not fourteen, and is not comparable with the table above. Engineering and law are much harder than the average subject — Nemotron scores 72.11 across all fourteen and 46.75 across these two. Read each table against itself.

At a 65,536-token generation cap

Every table above was measured with answers capped at 3,072 tokens, and that cap was doing more work than the models were. Re-measured at 65,536 — the point where no answer is truncated at all — the same models on the same questions score differently, and not by a little.

What it cost

This is not a like-for-like upgrade of the tables below. Both sets of numbers are real measurements of the same models; they differ in how much room the model was given to answer. The 3,072-cap tables are kept because the gap between the two is the result — a model that reasons at length was penalised by the old cap in a way that looked like a quality difference and was not.

Every subject, every model

The full breakdown behind the quality mean: fourteen MMLU-Pro subjects, 200 questions each, for every model swept at its default settings. Sorted by spread, so the subjects where models disagree most are at the top — which is where the averages stop being useful.

Engineering is the row to distrust. These sweeps ran at a 3,072-token cap, and raising it to 8,192 moved engineering by up to 31 points — Qwen3.6-27B goes from 60.0 here to 81.5. That row measures the harness more than the model. The two-subject table above uses the higher cap.

Throughput on the same questions

The speed each model sustained while producing those answers, measured from the server's own counters during the same sweep. Two tables rather than one because fourteen subjects against six models is already eight columns; accuracy and throughput share the rows, not the width.

Tokens spent per answer

The cost side, and the column that changes how the accuracy table reads. Two models scoring the same are not equally useful if one takes four times the tokens to get there.

How long each model takes to reach those scores is its own dimension — the same accuracy bought with four times the tokens is not the same result. That is the tokens-per-answer column in the quality table, and the quality versus speed post works through what it costs.

How to read this

Speed and quality come from different runs and are not a single score. A model can be fast and weak or slow and strong, and which matters depends entirely on what you are doing — the fastest model here is not the most accurate, and the most accurate is not close to the fastest.

Every figure traces to a JSON file in data/runs/ or data/subject-sweep/, produced by the harness described on the methodology page. Nothing on this page is typed by hand.