Local LLM Labs

A DGX Spark update made everything 3.4% faster

dgx-spark Qwen3.6-35B-A3BQwen3.6-27Bgpt-oss-120b

The DGX dashboard offered a system update described as “a driver update to improve handling of out-of-memory conditions and display stability”. I had benchmarked all three models on that machine earlier the same day, so for once I had a baseline before an update rather than after wishing I did.

Everything got faster. Twelve measurements, twelve improvements, mean +3.4%.

I do not think that is the whole story, and the reason why is more interesting than the number.

It was not a driver update

The dashboard’s description undersold it considerably. What was actually pending:

From To
Kernel 6.17.0-1018-nvidia 6.17.0-1029-nvidia
NVIDIA driver 580.159.03 580.173.02
CUDA toolkit config 13.2.75 13.3.29
Embedded Controller firmware 0x03000302 0x03000508
UEFI device firmware 0x0200980f 0x02009b0b
Everything else 336 packages

glibc, systemd, and a kernel bump carrying linux-modules-nvidia-580-open with it. Calling this a driver update is like calling a house move a change of address.

That matters for what any measurement can conclude. Even a perfectly clean before/after cannot attribute a change to the driver, because the driver is one of six things that moved.

The numbers

Same harness, same four workloads, same 32k context, same machine, same day. Throughput is end-to-end: total tokens generated over total wall time.

Model Scenario Before After Change
Qwen3.6-35B-A3B chat_short 50.2 52.1 +3.8%
chat_long_context 37.2 39.3 +5.5%
code_generation 50.4 52.8 +4.7%
summarization 23.0 23.3 +1.5%
Qwen3.6-27B chat_short 17.4 17.6 +1.6%
chat_long_context 13.6 13.8 +2.1%
code_generation 16.6 17.5 +5.3%
summarization 8.9 9.2 +4.0%
gpt-oss-120b chat_short 37.2 38.4 +3.1%
chat_long_context 30.7 31.6 +2.9%
code_generation 37.0 38.6 +4.3%
summarization 19.5 19.9 +2.3%

Range +1.5% to +5.5%, mean +3.4%.

Twelve out of twelve moved in the same direction. If the direction were random you would see that about once in four thousand attempts, so something real happened. Within-run standard deviation on these measurements is under 0.1 tok/s, which rules out ordinary jitter — but says nothing about variation between sessions, which turns out to be the crux.

Cold starts did not move at all: 27B 540s against 505–555 before, gpt-oss 495s against 465–570, the 35B 465s against 435–460.

The problem

The machine had been up for eleven weeks. The before-runs were taken on that machine. The after-runs were taken on a freshly booted one.

So there are two explanations and this data cannot separate them:

  1. The update made inference 3.4% faster.
  2. Rebooting after eleven weeks made inference 3.4% faster.

The second is not a stretch. Eleven weeks of allocation churn on a machine with one unified memory pool is exactly the setting where fragmentation, accumulated kernel state and page-table pressure could plausibly cost a few percent. A fresh boot would recover it, and the update would get the credit.

I cannot rerun the before-case: the old kernel and driver are gone, and even if I rolled them back the machine would have been freshly rebooted, which is the variable I am trying to isolate.

Total run time

Arm What ran Machine time
Qwen3.6-35B-A3B 4 scenarios · 5 repetitions each 4 min
Qwen3.6-35B-A3B 4 scenarios · 5 repetitions each 4 min
Qwen3.6-27B 4 scenarios · 5 repetitions each 11 min
Qwen3.6-27B 4 scenarios · 5 repetitions each 11 min
gpt-oss-120b 4 scenarios · 5 repetitions each 3 min
gpt-oss-120b 4 scenarios · 5 repetitions each 3 min
Total 35 min
Machine time for the six runs cited here — three models before the update and the same three after — from data/runs/.

About thirty-five minutes across both arms.

Twelve paired measurements for half an hour is why this post can say “twelve out of twelve moved in the same direction” rather than testing one model and calling it. The throughput harness is cheap enough that a single-run comparison has no excuse.

What that cheapness could not buy is the control. Both arms differ by a software update and a reboot, and re-running either one more times does not separate them — which is what the next section is about.

The experiment that would settle it

Re-measure in a few weeks, without rebooting.

If throughput drifts back toward the old numbers as uptime accumulates, the gain was the reboot. If it holds at the new level, the gain was the update.

That is a real prediction with a real way to be wrong, and I will publish it either way. It is also the sort of thing almost nobody tests, because it requires baselines from before an update you did not know was coming and the patience to wait weeks for the follow-up.

A hypothesis that did not survive

When the first model came back I thought I could see a mechanism. The 35B gained 3.8–5.5% on the decode-heavy scenarios and only 1.5% on summarization, which is prefill-dominated. That looked like a decode-path improvement.

The 27B does the opposite: summarization +4.0%, chat_short +1.6%. gpt-oss is mixed. Across all three there is no pattern by scenario type.

So the gain is broad and I have no mechanism for it. Writing that down is better than the alternative, which is retrofitting an explanation onto four numbers that happened to line up.

What else the update did, which is nothing

Worth recording for anyone weighing whether to take the downtime:

Eleven weeks of uptime is long enough that a reboot is its own risk: anything started by hand months ago and never made persistent does not come back. Worth auditing that before clicking update rather than discovering it after.

A note on the peer that ruined a run

The first sweep after the update failed on two of three models. Not a bug — a scheduled job on another project started at 21:00 UTC, mid-benchmark. The harness refuses to measure while another workload is on the GPU, so it quarantined the 27B result and declined to start the third model at all.

Two measurements lost, two wrong numbers prevented. That is the trade, and it is the right one: a benchmark taken against contention is not a slow measurement, it is a wrong one.

The irony is that I had built a usage dashboard that morning showing exactly when that job runs — 21:00 UTC, every day, confirmed across four weeks — and then started a sweep at 20:46 without checking it.

Setup

Source data: 96f13c64aa06303da530d411b43438259dc6fb8aca51197399ee050fe7f2cfc944049d1e