DGX Spark with 8 users: 2.9x tokens, 36% speed each

Every throughput figure published on this site — every number in the comparison table, every chat_short result in every post — is single-stream. One request in flight, batch of one. It is the right shape for “what does this feel like to use”, and it is what almost every local-LLM benchmark reports.

It is also a floor, not a ceiling, and until now nothing here said so.

A machine serving several requests at once produces more total tokens than one serving a single request, because the weights are read once per forward pass and that read serves every request in the batch. On a memory-bandwidth-bound machine, which the resident-size post showed this one is, that sharing is where the spare capacity lives.

So how much is there? Nobody publishes the curve for these machines. Here it is.

The measurement

One model, one quantisation, one server. A fixed 128-token prompt and a fixed 256-token output. The only thing that changes is how many requests are in flight simultaneously.

Qwen3.8-Flash-Next at UD-IQ1_S, llama.cpp with eight slots, five repetitions per level, zero failed requests, coherence gate 3/3. Standard deviations between 0.44 and 1.60 tok/s.

The two numbers point in opposite directions

Aggregate throughput nearly triples. Eight concurrent users get 89.8 tokens per second out of the machine against 31.0 for one. If you are running a batch job — grading essays, tagging documents, anything where nobody is waiting — that is 2.9× the work per hour for free.

Each individual user gets 36% of the speed. A single user sees 31 tok/s. That same user, sharing with seven others, sees 11.2. An answer that took ten seconds now takes twenty-eight.

A benchmark that reports only the first number flatters the machine. One that reports only the second — which is what this site has been doing — understates it. Both are true and they describe different jobs.

When this matters, and when it does not

The test is simple: does anything wait on an individual answer?

If nothing does, concurrency is close to free money. Two hundred questions to grade, a corpus to tag, a benchmark to run — the work is a queue of independent requests and only the total time matters. Here is the same 200-question batch at each level, using the measured 256-token answers:

Twenty-eight minutes becomes ten. Nobody is worse off, because nobody was waiting on question 137 in particular.

This is not hypothetical for us. Every MMLU-Pro sweep on this site is exactly that shape — 200 independent questions per subject. The quantisation comparison ran one arm at two concurrent requests and the other at six, and the six-slot arm finished 400 questions in 4.8 hours against 8.4. Same machine, same afternoon; the difference was how many questions it was allowed to work on at once.

If something does wait, concurrency does nothing for you. A single person chatting with a model is one request. There is no batch to share a weight read with, so the number that matters is 31 tok/s and the other rows are irrelevant. Worse, if that person shares the machine with seven batch requests, they get 11.2 tok/s — concurrency has actively made their experience worse.

So the practical form:

• Batch work — grading, tagging, extraction, evaluation, anything with a queue: run it wide. The machine has roughly three times more throughput than its single-stream number suggests.
• One interactive user — a chat, a coding assistant: the single-stream number is your number, and no amount of slots improves it.
• A shared machine doing both — a team’s inference box, or a Spark running services alongside experiments: the batch work will slow the interactive user down, measurably and roughly in proportion. That is a scheduling decision, not a hardware one.

Where the efficiency goes

The efficiency column is the interesting one. Two concurrent requests get 79% of perfect scaling. Eight get 36%.

That shape is what a bandwidth-bound machine looks like. Adding a second request is nearly free, because the expensive part — reading the weights — was already happening and the second request rides along on the same read. By eight requests the batch is large enough that the other costs, per-request KV cache traffic and attention over eight separate contexts, have grown to dominate, and each addition buys less.

The curve is still rising at eight. It has not flattened, so the ceiling is somewhere past where this machine could be tested — eight slots at 12,288 tokens each was what 121 GiB would hold alongside a 67.6 GiB model.

What this changes about the rest of the site

Every published throughput number is a single-user figure. For interactive use, that is the number you want. For anything serving more than one person or running a batch, the machine has roughly three times more capacity than those figures suggest.

That reframes some earlier conclusions. Nemotron 3.5 at 78 tok/s single-stream is not a machine doing 78 tokens per second — it is a machine doing 78 for one user and considerably more in total for several. The question “is a DGX Spark enough for my team” has a different answer than “is it fast enough for me”.

What this does not tell you

One model, one quantisation. A 67.6 GiB 1-bit MoE is not every model. A dense model, or one closer to the memory ceiling, will have a different curve — plausibly a worse one, since less spare bandwidth means less to share.

Short requests. 128 tokens in, 256 out. Long-context requests spend far more of their time in prefill, which parallelises differently, and eight simultaneous 8k prompts would compete for memory in a way eight 128-token prompts do not.

Slots were fixed at eight. llama.cpp allocates its slot count at launch, so this measures real concurrency at every level rather than queueing — but it does not tell you what sixteen would do, and this machine could not hold sixteen at a usable context length.

Nothing about latency distribution. The per-request figure is a median across five repetitions. A production serving setup cares about the tail, and this does not measure it.

Production services were stopped for the duration, to leave memory for eight slots. A machine also serving embeddings and a fast tier has less room and would show a shorter curve.

Total run time

Not recorded, and that is a gap in the harness rather than an omission here. bench/concurrency.py wrote its record without a duration_s field — the throughput harness has always stored one and this script did not — so the run behind this post has no measured wall clock to report.

It could be derived from the token counts and the throughput figures above, and that is exactly why it is not: a number computed from other numbers is an estimate, and putting one in a row labelled “machine time” beside figures that were actually measured is how a benchmark site stops being worth reading. The script now records the field, so the next concurrency run will have one.

The tool

bench/concurrency.py is in the repo. It reuses the throughput harness’s machine probe, backend detection and coherence gate, so a concurrency record carries the same provenance as any other run and refuses to measure a model that cannot answer three trivial questions correctly.

Two details in it are load-bearing. Each repetition builds fresh prompts with a new nonce, because otherwise the second round is served from the first round’s prefix cache and the curve measures caching rather than concurrency. And the slot count is a required argument that is recorded but cannot be verified — asking eight concurrent requests of a two-slot server measures queueing, and a record that did not state the slot count could not be read safely afterwards.