Running Muse Glimmer on a DGX Spark: five gotchas

Meta released Muse Glimmer on 10 August. NVIDIA’s launch post names the DGX Spark as a recommended platform for it, and publishes exactly one throughput number — measured on Blackwell Ultra, which is not a Spark.

I have a Spark. Between docker run and a server that answered a question correctly, there were five distinct failures. Four of them produce an error you can search for. The fifth produces a running server that returns HTTP 200 at a normal speed and is wrong about everything, which is the one worth reading about.

None of this is a complaint about the model. It is what a day-one release looks like on hardware that is not the hardware the release was tested on, and it is entirely reproducible — every command and error below is real.

Summary: what worked, what didn’t

Everything I tried, in order. Read the middle column before the first one — three of these configurations start a server that answers requests and is wrong about everything, which is a worse outcome than the ones that refuse to start.

Updated 2026-08-19: the depth-15 figures below were re-measured. The original run’s serving configuration was never recorded — the container had been started by hand, so the record could not show which drafter depth produced the fastest row. The re-run is the same model, weights, image and flags, and it reproduces the result: the two runs differ by less than noise (chat_short 20.97 ± 1.22 vs 22.39 ± 2.21 tok/s over five repetitions, t = 1.26). The numbers shifted slightly because they are a different draw, not because the earlier ones were wrong.

Row 11 is the answer. Rows 4, 5 and 7 are the reason this post exists — and rows 9 to 11 are why the post carries a correction: for a while I published row 10 as the answer, having blamed the crash on the draft depth rather than the batch budget that actually caused it.

What each working configuration gets you, since rows 8, 10 and 11 are the same model and weights with one flag changed between them:

Two scenarios shown because they bracket the range: short chat is the interactive case, code generation is the longest output and gains most from drafting. The other two land between them, and all four are in the benchmark post. Speculative decoding is lossless, so every row is the same output — the slower ones are just slower.

Memory, measured while serving row 10 (--gpu-memory-utilization 0.62 on the GB10’s 121.7 GiB of unified memory):

The BF16 baseline in row 8 is about 5 GiB lighter, that being the size of the DFlash draft model. Draft depth does not change the memory picture — the drafter is the same 5.1 GB model at depth 4 and depth 15; only how many tokens it proposes per pass differs.

Two things worth noting before you plan capacity. Loading the weights alone accounted for 60.7 GiB and took 396 seconds — most of the ten-minute cold start is reading the checkpoint off disk, not initialising the engine. And of the ~76 GiB total, only 10 GiB is KV cache: this model spends its memory on weights, which is the same fact that caps its speed.

That last point has a practical consequence. vLLM notes that the same box could allocate 33.25 GiB of KV cache instead of 10.1 if you asked it to, raising concurrency well beyond the measured 6.6× at 32k context. On a machine this size the constraint on serving more requests is not memory — it is that every one of them is competing for the same 273 GB/s of bandwidth.

1. The version number is lower than the version that fails

The model’s config.json declares MuseGlimmerForConditionalGeneration, and vLLM’s recipe says vLLM 0.27.0+. That is misleading. Native support lives on PR #51655, which is still unmerged, so no released version contains it.

You can check any image in about thirty seconds:

docker run --rm --entrypoint python3 <image> -c "
from vllm.model_executor.models.registry import ModelRegistry
print([a for a in ModelRegistry.get_supported_archs() if 'Glimmer' in a])
"

What that returns:

The image that works self-reports a lower version than two that do not. Searching by version number finds nothing; the support is carried by the tag. There are muse-glimmer-* tags for arm64 and x86_64, cu129 and cu130.

The habit this breaks is a reasonable one: read the requirement, check your version, upgrade until you meet it. That reasoning is sound and it fails completely here, because support for a new architecture arrives on a branch long before it arrives in a release, and a branch build is versioned from wherever it forked. “0.27.0+” was not wrong so much as it was answering a different question than the one I was asking.

The habit worth having instead costs thirty seconds: ask the image what it supports rather than what it is called. The registry check above is the whole test, it works for any model and any image, and it would have saved me several hours.

2. A 47 GB shard will not load on 128 GB of unified memory

With the right image, the first load stalled at 0/2 for eight minutes and the engine was killed:

Checkpoint size: 55.46 GiB. Available RAM: 35.88 GiB.
Loading safetensors checkpoint shards:   0% Completed | 0/2

vLLM streams weights one shard at a time, so peak host memory is roughly the largest single shard, not the whole checkpoint. Meta ships Muse Glimmer as two files, one of them 47 GB. Everything else on this machine arrives in small pieces:

The 35B and gpt-oss are larger overall and load without complaint. Total size is not the constraint; shard granularity is. The arithmetic is specific to unified memory: --gpu-memory-utilization 0.62 reserves its share of the same 121 GB pool the loader stages into, leaving 35.88 GB for a 47 GB shard.

The fix is to rewrite the checkpoint into smaller shards. Streaming tensor-by-tensor keeps peak RAM at about one output shard, so the split itself does not need 47 GB:

from safetensors import safe_open
from safetensors.torch import save_file
# plan shard membership from metadata (shapes + dtypes, no tensor loads),
# then materialise one output shard at a time.

Re-sharded 2 → 16 shards, largest 3.98 GB, it loads in about nine minutes alongside the always-on models with no memory pressure.

Large shards are not a mistake on the vendor’s part — fewer, bigger files download faster and are simpler to verify, and on a machine with discrete VRAM and separate system RAM the staging buffer comes out of a different pool, so nothing collides. The 47 GB shard is fine on the hardware it was tested on.

It stops being fine on unified memory, which is exactly the architecture the DGX Spark is sold for. That is the general lesson here and it will outlive this model: on unified memory, the loader and the accelerator are competing for one pool, so the largest shard is a hard constraint independent of how much total memory you have. A 128 GB machine that comfortably serves an 81 GB model can fail to load a 56 GB one.

If you package models, 4–5 GB shards cost you very little and make this class of failure impossible. If you run them, checking the largest shard against your free memory is a one-line test that will keep paying off as unified-memory machines get more common.

3. The fallback backend serves noise at a plausible speed

Before finding the right image, the obvious move is --model-impl transformers, since transformers 5.15 knows the architecture. It loads. It serves HTTP 200. It reports sensible token counts. And it is wrong:

prompt:  "The capital of France is"
output:  '\n_con_c\n\nThe\n'

The cause is visible in the startup log if you look for it:

Fused: input_layernorm (MuseGlimmerTextCenteredRMSNorm) -> GemmaRMSNorm (CustomOp)

A centered RMSNorm subtracts the mean; Gemma’s does not. Substituting one for the other corrupts activations in every layer. The same failure appears in BF16 and FP8, compiled and eager — --enforce-eager does not help, because the substitution happens at model construction, not in the compiler.

This is the dangerous one. Nothing about it looks like a failure: the server is up, latency is normal, throughput is plausible. A benchmark harness measuring tokens per second would have recorded four scenarios of publishable numbers from a model emitting gibberish.

I want to be blunt about how close this came to being published as a result.

I had a working endpoint, a benchmark harness, and four scenarios of clean, low-variance timings. Every instinct that normally tells you something is wrong — errors, warnings, latency spikes, missing tokens — said nothing, because from the outside a corrupted model and a working one are indistinguishable. Timing measures how fast tokens arrive, not whether they mean anything. The only reason those numbers did not end up on this site is that I pasted a question into the endpoint by hand before running the benchmark.

So the harness now refuses to record a run until the model has answered a few questions with unambiguous answers. Three prompts, two of three must pass, matched case-insensitively anywhere in the reply so a model that thinks out loud still counts. It costs a couple of seconds per run and it would have caught all three of the broken configurations above in the first one.

If you take one thing from this post: check that the output means something before you trust any number measured alongside it. Not just for a new model — for a new image, a new quantization, a new flag. Every one of those changes the kernels that run, and a wrong kernel is silent.

4. Both quantized paths are dead ends on this chip, for now

FP8 halves the bytes per token, and RedHatAI’s FP8-block build is 34.4 GB with a 28.2 GB largest shard, which would sidestep problem 2 as well. It dies during weight processing:

RuntimeError: Assertion error (deepgemm-src/csrc/apis/layout.hpp:60):
Unknown SF transformation

DeepGEMM has no scale-factor layout for SM121, the GB10’s compute capability. Setting VLLM_USE_DEEP_GEMM=0 gets past it — and produces a server that returns '<|eom|>_cassistant_'. It converts a loud crash into silent corruption, which is worse.

NVFP4 is smaller still, but it is the same FP4 family that already requires VLLM_MXFP4_BACKEND=marlin on this machine to stop gpt-oss corrupting its first token. I did not pursue it.

Quantization is worth revisiting, but not yet, and the reason is narrower than “FP8 is broken on this chip”. The GB10 reports compute capability 12.1, and the kernel libraries dispatch on architecture: DeepGEMM has layouts for the architectures it was built against and no entry for this one. That is a lookup-table gap, not a hardware limitation, and it closes on its own the moment someone adds the case.

What I would want before trying again is simply a build whose FP8 path has been exercised on SM121 — the NVIDIA NGC images are the obvious candidate, since they exist precisely to carry this kind of chip-specific fix, and they already carry one for gpt-oss’s MXFP4 path.

The part I would not repeat is VLLM_USE_DEEP_GEMM=0. Turning off an accelerated path to get past an assertion is a reasonable instinct, and here it traded a crash I could see for corruption I could not. An environment variable that makes an error disappear has not necessarily fixed anything, and on this occasion it made the situation strictly worse.

5. The drafter needs batch space, and without it the error blames the wrong thing

Muse Glimmer ships its own DFlash drafter, and the published recipe says:

--speculative-config '{"method":"dflash","model":"...-assistant","num_speculative_tokens":15}'

Run that as-is and the engine dies during init:

RuntimeError: CUDA error: device-side assert triggered

The cause is in the warning vLLM prints on the way past, which I quoted here for weeks while misreading it:

max_num_scheduled_tokens is set to 2048 based on the speculative decoding settings. [...] Consider increasing max_num_batched_tokens [...] or decrease num_speculative_tokens.

Two remedies, and they are not equivalent. DFlash needs max_num_seqs × (1 + num_speculative_tokens) of batch space to stage its draft inputs — with 256 sequences and 15 draft tokens, 4096. Left to itself the scheduler caps at 2048, the draft input tensor overruns, and the position values become garbage; the assert is downstream of that. Lowering the draft count shrinks the requirement until it fits. Raising the batch budget fixes it properly and keeps the speed:

--max-num-batched-tokens 8192
--speculative-config '{"method":"dflash","model":"...-assistant","num_speculative_tokens":15}'

At depth 15 with that headroom it runs clean, at a mean acceptance length of about 3.4:

Speculative decoding is lossless — the full model verifies every drafted token — so all three columns are the same output at different speeds.

One caveat on code_generation: at depth 15 its repetitions ranged from 24.3 to 31.9 tok/s, a standard deviation of 3.9. Deeper drafting makes throughput more sensitive to how predictable the output is, so treat that 25.93 as the softest number in the table.

The 4.3 tok/s baseline is not a tuning failure. 4.3 tok/s × 59.6 GB is about 256 GB/s of weight traffic, against roughly 273 GB/s of memory bandwidth: the model was already running at about 94% of what the machine can move. Decode stayed flat at 4.3 whether the prompt was 128 tokens or 16,384, with KV cache usage at 0.2% throughout. Speculation does not raise that ceiling — it stops spending a full sweep of the weights on every single token.

This is the number that should change how you choose models for a machine like this one, so it is worth being precise about what it says.

The Spark was not running Muse Glimmer badly at 4.3 tok/s. It was running it at essentially the speed physics allows: one token requires reading every weight, 59.6 GB of them, and the machine can move about 273 GB/s. No flag, no image, no amount of tuning moves that. On bandwidth-limited hardware, single-stream decode speed is a property of bytes-per-token, not of how good the implementation is — which is why decode stayed pinned at 4.3 whether the prompt was 128 tokens or 16,384.

That gives you exactly three levers, and it is worth knowing which one you are pulling:

• Fewer bytes per token. Quantization. Halve the weights, roughly double the speed — the reason a 27B served at FP8 outruns a 30B at BF16 despite being a comparable model.
• Fewer bytes read per token. Mixture-of-experts. This is why gpt-oss-120b, at 117B parameters, is faster than a dense 30B: it only reads the experts it activates.
• More tokens per read. Speculative decoding, which is what DFlash is doing above.

Only the third is free. The first two trade capacity or precision for speed; speculation gives you 2–3× and verifies every token against the full model, so the output is unchanged. If you are choosing a model for a bandwidth-limited box, check whether it ships a drafter before you check its benchmark scores — a model with one will beat a nominally faster model without one, on your hardware, by a margin larger than most quality differences.

The configuration that works

docker run -d --rm --name vllm-muse-glimmer-30b \
  --gpus all --ipc=host --shm-size=16g -p 8008:8008 \
  -v /path/to/Muse-Glimmer-30B-resharded:/model:ro \
  -v /opt/vllm/cache:/root/.cache/huggingface \
  --entrypoint vllm vllm/vllm-openai:muse-glimmer-arm64-cu130 \
  serve /model --served-model-name meta-models/Muse-Glimmer-30B \
  --port 8008 --host 0.0.0.0 --max-model-len 32768 \
  --tensor-parallel-size 1 --gpu-memory-utilization 0.62 \
  --max-num-batched-tokens 8192 \
  --speculative-config '{"method":"dflash","model":"meta-models/Muse-Glimmer-30B-assistant","num_speculative_tokens":4}'

Cold start is about ten minutes: sixteen weight shards plus a 5.1 GB drafter.

One rough edge remains. This build exposes no muse_glimmer reasoning or tool-call parser, so the model’s channel-scoped reasoning arrives raw in content — replies can open with to=self and the model thinking out loud before answering. Strip it, or match on the answer rather than assuming the whole reply is the answer.

Total run time

About an hour of successful measurement, which badly understates what this post cost.

That hour is the configurations that worked. It does not count the images that would not start, the 47 GB shard that would not load, or the backend that served fluent nonsense at a perfectly normal speed — that last one being the expensive kind of failure, because it produces a plausible number and no error.

The measuring is cheap here. Finding a configuration worth measuring is the work, and none of it appears in the figure above.

Should you do this today?

If you want to use the model: wait. PR #51655 will merge, the NGC images will pick it up, and most of this post becomes history — the pre-release tag, the re-sharding, the missing parsers. At that point it is an ordinary vllm serve against an ordinary image, and the only thing left from this list is the draft depth. There is no capability here you cannot have in a few weeks for none of the effort.

If you want to understand the machine: do it now, or do the equivalent with whatever ships next. Nothing in this post is specific to Muse Glimmer. Shard granularity against unified memory, kernel dispatch gaps on a new compute capability, a fallback path that is wrong rather than absent, a vendor default that crashes on your hardware — those recur with every new architecture, and you learn them by hitting them, not by reading release notes.

If you are running a DGX Spark in particular: the bandwidth arithmetic is the durable thing. A 60 GB model at BF16 will give you about 4 tok/s on this machine and no configuration changes that. Decide whether you can quantize it, whether it is sparse, or whether it ships a drafter — before you buy into it, not after.

And whatever you run: check that it can answer a question you know the answer to, before you believe anything else it tells you.

Source data: d994a07defc0d4fa66cd5bd9e6223a2d9555