Muse Glimmer on a DGX Spark: 4 tok/s, or 23 with a flag

Meta’s Muse Glimmer is a dense 30B model built for always-on local agents, and NVIDIA’s launch post names the DGX Spark as a recommended platform for it. So: how fast is it on one, and is it fast enough to sit inside an agent loop?

The short answer is 4.3 tokens per second, which is not fast enough for anything interactive — and then 23.3, which is, from a single flag that costs nothing in output quality. The gap between those two numbers is the whole post.

Setup

All three rows are the same weights on the same machine. Only the speculative-decoding configuration differs — off, drafting four tokens ahead, and drafting fifteen — which is what makes the comparison clean.

Getting to the first row took most of a day. On the day of release no shipping vLLM contained the architecture at all — not NVIDIA’s NGC images, not the current stable release — and the checkpoint’s 47 GB shard could not be staged into memory on a 128 GB machine. The serving stack here is a pre-release image built from an unmerged pull request, running a locally re-sharded copy of the weights. The companion post covers all of that in detail, including a configuration that served fluent-looking gibberish at a perfectly normal speed.

Everything below was measured after the model demonstrably answered questions correctly. Given how easy it was to produce numbers from a broken model, that is not a formality.

Results

Updated 2026-08-19: the depth-15 figures below were re-measured. The original run’s serving configuration was never recorded — the container had been started by hand, so the record could not show which drafter depth produced the fastest row. The re-run is the same model, weights, image and flags, and it reproduces the result: the two runs differ by less than noise (chat_short 20.97 ± 1.22 vs 22.39 ± 2.21 tok/s over five repetitions, t = 1.26). The numbers shifted slightly because they are a different draw, not because the earlier ones were wrong.

chat_short

A 128-token prompt and 256 out: almost pure decode, and the closest thing here to what typing at the model feels like. The baseline’s 4.3 tok/s is roughly reading speed for a slow reader — technically usable, unpleasant in practice. At 23.3 it stops being the thing you notice.

Prefill is not compared in this post. Speculative decoding emits several tokens per stream delta, so time-to-first-token stops meaning what it usually means and the harness records it as null rather than reporting a number it cannot stand behind. For reference, the baseline measured 297 ms here.

chat_long_context

8k in, 256 out — the RAG-shaped case, where a real prompt of retrieved context precedes a short answer. This is the smallest speedup of the three decode-heavy scenarios (3.37×) for a simple reason: a meaningful share of the wall clock is prefill, and speculation does nothing for prefill. The baseline spent 3.8 s before the first token.

code_generation

512 in, 1024 out: the longest generation in the suite, and the best case for speculation at 6.06×. Long outputs are where drafting compounds — every accepted token is a full weight sweep the machine did not have to do, and across 1024 tokens that adds up.

Worth noting for agentic use specifically: this is the shape that matters, and it is the shape that gains most.

summarization

16k in, 128 out — the worst case for a bandwidth-limited machine, and the smallest speedup at 2.80×. With a 16,384-token prompt and only 128 tokens of output, prefill dominates the wall clock, and prefill is unaffected by drafting. The baseline waited 7.6 s before the first token.

If your workload looks like this — long documents in, short answers out — speculative decoding is worth much less to you than the headline figure suggests.

What the numbers say

End-to-end throughput, median of 5 repetitions. Both runs are the same weights, image and machine; only the speculative config differs.

The bottleneck is memory bandwidth, and the baseline was not missing it by much.

Decoding one token from a dense model requires reading every weight. At BF16 that is 59.6 GB, and 4.3 tokens per second works out to about 256 GB/s of weight traffic against roughly 273 GB/s of memory bandwidth on the GB10. The baseline was running at around 94% of what the machine can physically move. There was no configuration left to find.

The clearest evidence is that decode did not care about the prompt at all: it sat at 4.3 tok/s whether the input was 128 tokens or 16,384, with KV cache occupancy at 0.2% throughout. The weights dominate so completely that the attention cache is a rounding error. Once you have seen that flatness, the number stops looking like a tuning problem and starts looking like a division.

Which is why speculative decoding is the whole answer here. It does not raise the ceiling — the machine still moves 273 GB/s — it stops spending a full sweep of the weights on every single token. At a mean acceptance length of about 3.4, one pass over the weights yields three or four tokens instead of one, and the measured 2.8–6.1× falls out of that. The variation across scenarios is just how much of each one is decode: code_generation is nearly all decode and gains 6.06×, summarization is mostly prefill and gains 2.80×.

Acceptance is not constant either. On highly predictable output the drafter reached a mean acceptance length of 9.6 — nearly ten tokens verified per pass, which is only reachable at all because the draft depth is 15.

Two things did surprise me. The first is that the drafter is not an optional extra on hardware like this — it is the difference between a model you can put in an agent loop and one you cannot, and it ships in the same repository as the weights.

The second is how easily I misdiagnosed it. Meta’s recipe specifies num_speculative_tokens: 15; run that as published and the engine dies with a CUDA device-side assert, which reads exactly like a hardware incompatibility on a new chip. It is not. DFlash stages its draft inputs in the scheduler’s token budget and needs max_num_seqs × (1 + num_speculative_tokens) of it — 4096 here — while the scheduler defaults to 2048. The tensor overruns and the positions become garbage. Dropping the draft count shrinks the requirement until it fits, which is why depth 4 appeared to be the fix; raising --max-num-batched-tokens fixes the actual problem and keeps the throughput. The two remedies look equivalent from the error message and differ by up to 2× in speed.

But is it any good?

Speed is worth nothing on its own, and this post would be misleading if it stopped at throughput. MMLU-Pro, 200 questions in each of fourteen subjects — 2,800 in total, 5-shot chain-of-thought, greedy — the same harness, questions and answer-extraction rules used for the other three models on this box. How the grading works is written up separately.

Muse Glimmer scores 74.9%, statistically tied with gpt-oss-120b for third.

So it is comfortably behind both Qwens and indistinguishable from a model four times its size. The aggregate still hides the more useful story, which is that it is not uniformly weaker — it is competitive in most subjects and falls off badly in a few:

The vs worst column is the one that reframes it. Muse Glimmer is below all three of the others in five of fourteen subjects — chemistry, computer science, economics, law and physics — and it beats the weakest of them in eight. Engineering is its standout: ten points clear of the worst model and within 1.5 of the best, in the one subject where the 27B gets displaced at all.

Law is the real weakness, and it is severe. 49.0% against the 27B’s 75.5% — a 26.5-point gap, the widest single-subject deficit between any two models on this machine, on a ten-option format where chance is 10%. Nothing about the sample size explains that away.

The correction is worth dwelling on, because the original post named the number it distrusted and that instinct was right. It said of physics: “the number I trust least and would want to re-run. A model that ties the field on math and then drops to 70% on physics is a strange shape, and at n=20 that is six wrong answers. It may be real. It may be six unlucky draws.”

It was six unlucky draws. At 200 questions physics came back at 80.0% — an ordinary result for this model, in line with chemistry and psychology. History did the same thing, moving 31 points. Both were noise wearing the shape of a finding, and the way to tell was not judgement but a larger sample.

Per-subject figures are now 200 questions each, which is about ±3 points at one standard error rather than ±11. Single-point differences within a row are still nothing; the ten- and twenty-point gaps are now real.

Total run time

About an hour across three configurations, and the distribution is the point: the no-drafter run took 40 minutes and the depth-15 run 8.

The benchmark’s own duration is the finding. A five-fold difference in wall clock for identical work is the same 2.8-6.1x speedup the table reports, arriving before any number is computed.

What to run

There is no buying decision in this post — it is one model on one machine, in two configurations — so the recommendation is about how to run it.

Turn the drafter on. It is 2.1–3.2× faster and the output is unchanged, because the full model verifies every drafted token before it is emitted. This is not a speed-versus-quality trade; there is no cost to weigh. The only reason not to enable it is the small amount of extra memory the draft model occupies, which on a 128 GB machine is not a consideration.

--speculative-config '{"method":"dflash","model":"meta-models/Muse-Glimmer-30B-assistant","num_speculative_tokens":4}'
--max-num-batched-tokens 8192

Give the drafter room, then use the recipe’s depth of 15. Set --max-num-batched-tokens above max_num_seqs × (1 + num_speculative_tokens) — 8192 covers the default 256 sequences at depth 15 comfortably. Without it you get an assert that looks like a hardware problem; with it, depth 15 is 1.3–2.0× faster than depth 4 at a higher acceptance length (3.4 against 3.0), so the lower depth costs throughput for nothing.

Judge the model on the DFlash row, not the baseline. 4.3 tok/s is what this machine does with 59.6 GB of dense BF16 weights and no drafting; it says more about bandwidth arithmetic than about Muse Glimmer. 23.3 tok/s is what the model delivers configured as its authors intended — faster than the dense 27B on this box.

But do not read that as “better than the 27B”. It is faster and it is the only model here that can read an image; it is also 7.6 points behind on MMLU-Pro, and that gap is statistically significant where the 27B-vs-35B gap in the four-model comparison was not. This is a speed-and-modality choice, not a quality one.

Where the deficit nearly vanishes: engineering (−1.5 and second of four), biology (−2.0), business (−3.0) and math (−3.0). Where it does not: law (−26.5), and to a lesser degree history, other, philosophy, economics and computer science, all −10 or worse.

Two of those deserve flagging because they cut against the obvious reading. “STEM-shaped work is fine” is only half true — engineering and math are fine, but computer science is one of its five worst subjects at −10.0, so this is not a model to reach for on coding-adjacent reasoning just because it does well on engineering. And if your work touches law, pick the 27B and accept the slower tokens; that is the one subject where the gap is large enough that no amount of speed compensates.

If your workload is long-in, short-out, temper expectations. The 2.80× on summarization is the honest figure for document processing, against 6.06× on code generation. Speculation helps decode, and that workload barely decodes. Note also that code_generation was the noisiest scenario at depth 15 — repetitions ranged 24.3 to 31.9 tok/s — because deeper drafting makes throughput more sensitive to how predictable the output is.

Revision history

Source data: d994a07defc0d4fa66cd5bd9e6223a2d9555