Measured benchmarks for running large language models on hardware you own. No vendor benchmarks, no reprinted spec sheets — just numbers measured on machines sitting in my office.
-
Qwen3.8-27B at a 65,536-token cap: the one model that needed it
Fourteen MMLU-Pro subjects re-measured with the answer budget raised from 3,072 to 65,536 tokens. 79.75 to 83.54, and unlike every other model in the campaign this one was still gaining at the top of the range.
-
Muse Glimmer at a 65,536-token cap: the model is its own control
Fourteen MMLU-Pro subjects re-measured with the answer budget raised from 3,072 to 65,536 tokens. Six subjects whose answers grew gained 4.4 points; eight whose answers did not gained 0.4.
-
Qwen3.6-27B at a 65,536-token cap: 85.96, the highest score on this site
Fourteen MMLU-Pro subjects re-measured with the answer budget raised from 3,072 to 65,536 tokens. The mean goes 82.54 to 85.96, physics reaches 95.5, and the cap stopped mattering at 8,192.
-
Qwen3.6-35B-A3B at a 65,536-token cap: +3.5 points, and 8,192 would have done it
Fourteen MMLU-Pro subjects re-measured with the answer budget raised from 3,072 to 65,536 tokens. The mean goes 80.79 to 84.32, engineering goes 52.0 to 82.5, and every point of it had already arrived by 8,192.
-
gpt-oss-120b does not care about the answer budget
The same 21x increase in generation cap that moved Nemotron 3.5 by 7.6 points moves gpt-oss-120b by 0.29. Fourteen subjects: six up, six down, two unchanged.
-
Nemotron 3.5 at a 65,536-token cap: +7.6 points
Fourteen MMLU-Pro subjects re-measured with the answer budget raised 3,072 to 65,536 tokens. Engineering goes 45.0 to 81.0, and the mean goes 72.1 to 79.8.
-
DGX Spark with 8 users: 2.9x tokens, 36% speed each
One request to eight on a GB10: total output nearly triples, and each individual user waits nearly three times as long. The whole curve, measured.
-
1-bit vs 3-bit on a DGX Spark: 27% faster, no cost
Qwen3.8-Flash-Next at two quantisations, 400 MMLU-Pro questions per arm. 16 GiB lighter, and the accuracy deltas point in opposite directions.
-
DeepSeek-V4-Flash on a DGX Spark: 17 tok/s at 3-bit
291B parameters, 8B active, squeezed into 121 GiB. Slower than a model a quarter its size: throughput tracks resident size, not active size.
-
It was the reboot, not the DGX Spark update
Twelve fresh pairs, a kernel-only update and an accidental outage supplied the missing control. The software changed nothing; eleven weeks of uptime did.
-
Qwen3.8-Flash-Next on a DGX Spark: 23.5 tok/s
The first Qwen4-architecture model on a GB10, day one. It needs an unmerged pull request to load, and an open crash report names this exact chip.
-
MMLU-Pro engineering mostly measures truncation
Six models, two subjects, three generation caps. Raising the budget from 3072 to 8192 moved Qwen3.6-35B from 52.0 to 83.0, and did nothing for law.
-
Two models at once on a DGX Spark: the fast one pays
A 121 GiB machine holds two models comfortably. Running both costs one of them 65% of its throughput and the other nothing at all.
-
Nemotron 3.5 on a DGX Spark: 139 tok/s
2.7x faster than anything else measured on this GB10, and one flag accounts for nearly all of it. It also serves a real 1M-token context on 121 GiB.
-
What MMLU-Pro questions actually look like
Every quality number on this site comes from MMLU-Pro. What the questions are, where they come from, and why a random guess is worth 11.26%, not 10%.
-
Qwen3.8-27B on a DGX Spark: half the tokens, 2.8 down
The Qwen3.6-27B successor answers 2,800 MMLU-Pro questions in half the wall clock and scores 2.79 lower, losing on law and gaining on chemistry.
-
gpt-oss ignores its reasoning_effort setting
20,000 MMLU-Pro questions, two models, three effort levels each. gpt-oss-120b moves 0.3% in generation volume; Muse Glimmer moves 80% on the same harness.
-
Four models, fourteen MMLU-Pro subjects, one DGX Spark
11,200 questions and 35.8 GPU-hours on a GB10. The aggregate ranking inverts completely in one subject, and the model tying for third never wins one.
-
Muse Glimmer on a DGX Spark: 4 tok/s, or 23 with a flag
Meta's 30B runs at 4.3 tok/s, about 94% of the machine's memory bandwidth. Turning on the drafter it ships with makes it 2.8-6.1x faster, losslessly.
-
Running Muse Glimmer on a DGX Spark: five gotchas
The image that works reports a lower version than two that fail, a 47 GB shard will not load on 128 GB, and the fallback backend serves fluent noise.
-
A DGX Spark update made everything 3.4% faster
Twelve measurements across three models, all improved after a kernel, driver, firmware and 336 packages. The machine had also been up eleven weeks.
-
gpt-oss-120b on a DGX Spark: ~37 tok/s, four gotchas
A 117B model at ~37 tok/s on a desk-side box. Getting there: a 133 GB download that should be 57 GB, a mandatory backend flag, and two false readings.
-
Four models on a DGX Spark: size predicted nothing
MMLU-Pro at 2,800 questions each on one GB10. The four split into two clear quality tiers, and parameter count from 27B to 117B does not predict which.
-
Three models on a DGX Spark: fewest bytes still loses
Qwen3.6-27B, Qwen3.6-35B-A3B and gpt-oss-120b measured identically on one GB10. The 117B reads the fewest bytes per token and still loses to a flag.
-
lm-eval scored a good model at 13% on MMLU-Pro
A stop sequence fires inside a reasoning model's own chain-of-thought, truncating every answer before it arrives. One flag moves 13.3% to 86.7%.
-
35B MoE vs 27B dense on a DGX Spark: 3x the speed
Qwen3.6-35B-A3B against Qwen3.6-27B on a GB10: decode, prefill, time-to-first-token, cold starts and MMLU-Pro. The MoE wins 3x and ties on quality.