Two models at once on a DGX Spark: the fast one pays

A DGX Spark has 121 GiB of unified memory. Nemotron 3.5 at NVFP4 is 21 GB of weights; Qwen3.6-27B at FP8 is about 27 GB. Both fit at once with room to spare, which makes an obvious question worth asking: can you serve two models simultaneously and get two models’ worth of work out of the machine?

You can serve them. You do not get two models’ worth of work. And the way the cost is distributed is not the way I expected — it is not even close to the way I expected.

The measurement

Three arms, one pair of containers:

1. Nemotron under load, Qwen resident but serving nothing
2. Qwen under load, Nemotron resident but serving nothing
3. Both generating simultaneously

The first two matter more than they look. Measuring each model alone with the other one still resident means all three arms have identical memory allocation, so the only thing that changes in arm 3 is traffic on the memory bus. Benchmarking each model on an empty machine and comparing to a shared one would confound bandwidth contention with reduced KV cache, and the difference between those two explanations is the whole point.

Same scenario throughout — chat_short, a 128-token prompt and 256 out, five repetitions, near-pure decode.

One number below needs its context stated up front. The day-one post headlines Nemotron at 139 tok/s; here it starts at 77.42. Both are right. That post measured 78.35 without speculative decoding and 139.42 with the drafter enabled, and every arm here ran without it, so 77.42 is the comparable baseline. Speculative decoding was left off deliberately: it spends compute to save bandwidth, which is exactly the variable under test, and letting one model draft while the other saturates the bus would confound the thing this post is trying to isolate.

Nemotron 3.5 NVFP4

Qwen3.6-27B FP8

End-to-end throughput, chat_short, tok/s — higher is better. Both models are resident on the GPU in every column; only what the other model is doing changes.

Medians across five repetitions; ± is standard deviation. Neither of these is a marginal result.

The slow model pays nothing. The fast model pays everything.

Why it happens

The intuition that fails here is “two processes share a resource, so they each get about half of it.” Memory bandwidth is one shared resource — roughly 273 GB/s on this machine — but the two models have completely different relationships to it.

The paragraphs that follow are arithmetic, not measurement. The 273 GB/s is GB10’s specified bandwidth rather than something this harness measures, and the per-token weight traffic is inferred from parameter counts and quantisation. The throughput figures are measured; the roofline they are compared against is calculated, and a calculated ceiling is always optimistic — real sustained bandwidth on any machine runs below spec.

Qwen3.6-27B is a dense 27B at FP8. It reads its entire weight set for every token it generates: about 27 GB. Against 273 GB/s that puts its ceiling near 10 tok/s, and it measured 7.87 alone — roughly 78% of a ceiling no implementation actually reaches. There is very little slack in that number. Its demand is close to inelastic — it needs those bytes, it will wait for them, and it finishes when it finishes.

Nemotron is a 30B mixture-of-experts with roughly 3B parameters active per token at NVFP4. It reads a small fraction of its weights per token — about 2.1 GB, taking the 21 GB of weights at the 3-in-30 active ratio — which puts its own ceiling near 130 tok/s. At 77.42 it was running at roughly 59% of that, meaning a substantial share of its per-token time is compute and launch overhead rather than waiting on memory. It had headroom.

Put them together and the model with headroom is the one that loses it. Qwen takes the bandwidth it was always taking. Nemotron gets what is left, and what is left is a third of what it had.

The general form: on a bandwidth-saturated machine, a co-tenant does not take half your throughput. It takes your slack. A model with no slack is immune. A model whose speed comes from having slack is not.

What this rules out

I had been sketching a majority-voting setup — run the same question through several models, take the answer they agree on — on the assumption that a fast MoE model could ride along beside a slow dense one for free. That assumption is dead. Pairing them gets you the dense model’s latency and a two-thirds haircut on the fast one. The vote finishes when the last model finishes, so you would be paying 65% of Nemotron’s speed to gain nothing on wall clock.

Concurrent multi-model serving on this hardware only makes sense between models that all have bandwidth headroom — which on this fleet means the A3B-class MoE models, and nothing else. Two dense models would simply queue behind each other on the bus.

Getting them both running at all

Two failures worth recording, because neither is obvious and both cost me a run.

Start them sequentially, not together. My first working attempt launched both containers back to back and Nemotron died:

ValueError: No available memory for the cache blocks.

while Qwen’s log showed it had profiled against a machine that already had another engine allocating into it:

Free memory on device (78.1/121.69 GiB) on startup

--gpu-memory-utilization is a fraction of total memory, and vLLM subtracts what it observes as already consumed at startup. Two engines profiling at the same moment each account for the other’s half-finished allocation, and the arithmetic is meaningless. Bring the first one fully up, let it settle, then start the second.

Budget both utilizations against the same total, and leave real headroom. 0.32 for Nemotron and 0.40 for Qwen sums to 0.72 of 121.69 GiB — about 88 GiB against roughly 102 GiB free with the machine’s other services resident. At 0.28 Nemotron’s ~19.6 GiB of NVFP4 weights plus activation and CUDA-graph memory left almost nothing for KV even uncontended.

Total run time

Eight minutes across four runs — each model alone, then each under contention.

The measurement is trivially cheap; what it cost was memory arithmetic. Getting two models resident on 121 GiB at once took several failed attempts that produced no run files, and the section above on starting them sequentially is what came out of that rather than out of these eight minutes.

What this does not tell you

This is one pair. A fast MoE against a slow dense model is the most extreme case — maximum slack meeting maximum saturation. Two MoE models would divide things differently, and I have not measured that.

These are single-stream measurements. Each model had one request in flight. Real serving means batching, which changes the arithmetic: a batched dense model amortises its weight reads across concurrent requests and stops being bandwidth-bound in the same way.

Nothing here is about quality. Both models were measured on throughput alone, in the same configuration the rest of this site uses, so the numbers are comparable to every other speed figure published here — with the exception of the contended runs, which are marked in their own records as not comparable and are excluded from the fleet tables for that reason.

Source data: 791e7ba83f094ad10655ca80af5678632143f1b869a9ba82