1-bit vs 3-bit on a DGX Spark: 27% faster, no cost

The DeepSeek post established that on this machine throughput tracks how much of a model is resident, not how much is activated. Shrinking Qwen3.8-Flash-Next from 83.8 GiB to 67.6 GiB — the same model, same build, same flags, only the quantisation changed — took throughput from 23.54 to 29.81 tok/s. Resident size to 0.807×, speed to 1.27×.

That leaves the obvious question. Three bits per weight down to roughly one is an aggressive squeeze. What does it cost?

The answer, at the sample size I can afford: nothing I can detect.

The measurement

MMLU-Pro, engineering and law, 200 questions each per arm — 800 questions in total. Both subjects at an 8192-token generation cap, which the generation-cap post showed is where engineering stops being truncated. Engineering is the subject most sensitive to a model’s reasoning holding together; law is the control that barely moved under any other intervention.

Why this reads as no effect rather than a small loss

Both differences sit inside the interval that 200 questions permits. That alone would only mean “we cannot resolve it.”

The stronger signal is the direction. Engineering fell 4.5 points; law rose 2.5. A real degradation from crushing weights to one bit would push both subjects the same way — it would show up as the model reasoning less reliably everywhere, not as a loss on one subject paid back on another.

Opposite signs across two subjects is what scatter looks like. It is the same reasoning that settled the reboot question: when a real effect was present, twelve out of twelve measurements moved together; when it was absent, they split.

What you get for it

27% more throughput and 16.2 GiB back. On a 121 GiB machine that 16 GiB is not a rounding error — it is the difference between two concurrent slots and six, or between running alongside your other services and not.

The honest limit

±9 points is a wide band. This rules out a large degradation. It does not rule out a small one. Unsloth’s own retention figures put UD-IQ1_S at 80.2% of full quality against UD-Q3_K_XL’s 90.4% — a gap that, if it translated directly to MMLU-Pro, would sit right at the edge of what 200 questions can resolve.

The claim is “no measurable cost at this sample size”, not “no cost”. Resolving a five-point difference with confidence needs roughly four times the questions, which is roughly four times 13 hours of GPU time.

Two subjects, not fourteen. Engineering and law were chosen as the extremes of answer length. The other twelve were not measured at these quantisations.

One model. Qwen3.8-Flash-Next is a sparse MoE with about 6B active parameters and a 51B N-gram lookup table. How aggressive quantisation treats a lookup table of that size is not something to generalise from one model, and it is a plausible reason this particular architecture survives 1-bit better than a dense model would.

Total run time

Thirteen hours for 800 questions — two subjects, 200 questions, two quantisations.

Per question that is far more expensive than any other sweep on this site, for two reasons. The 8,192-token cap lets answers run nearly three times longer than the 3,072 default most sweeps used, and llama.cpp on a 67.6 GiB model is slow enough that the smaller arm still only manages about 30 tok/s.

The gap between the arms is the finding restated as time: the 3-bit model took 8.4 hours and the 1-bit model 4.8 for identical work. That is the 27% throughput difference compounded over 400 questions each.

It is also the reason the sample stops at 800 questions. Resolving a five-point accuracy difference needs roughly four times the sample, which is another fifty-odd hours for a result that would still be a single model at a single context length.

A note on the three attempts that failed first

This sweep took four tries. The first three produced no data at all, and the reason is worth writing down because it was not what I assumed.

I lost 4.7 hours to a run where every category failed with Session is closed, and diagnosed it as a client timeout. It was not. The server log showed 384 requests killed by “Context size has been exceeded”. llama.cpp reports n_ctx_slot = 16384 while also reporting kv_unified = true — and when the KV cache is unified, --ctx-size is a total divided across slots. Four slots meant about 4,096 tokens each, against the ~11,000 an MMLU-Pro request needs once a five-shot prompt and an 8192-token answer are counted.

Acting on that finding is what broke the machine: I raised the context to 49152 without checking it against the 14 GiB of headroom I had measured myself an hour earlier, and drove the box into memory exhaustion severe enough that sshd could accept connections but not fork a session. It needed a physical power cycle.

What the failures did not do is produce numbers. Every failed category recorded accuracy: null. That is the result of a fix made earlier in this campaign, after the harness was caught reporting a previous run’s score when lm-eval had written nothing — a three-bit GGUF’s engineering figure that was really an eight-thousand-token-cap run read back off disk. The failed records are committed alongside the successful ones, nulls and all.

A benchmark that fails loudly is worth more than one that fails quietly, and that is not a platitude here: without that fix, this post would have reported two fabricated accuracy figures and I would not have known.

Source data: 028ad1c3369daa8df7c97943