It was the reboot, not the DGX Spark update

On 9 August this site published a post about a DGX Spark update that made everything measurably faster. Twelve measurements, twelve improvements, mean +3.4%. The title conceded the problem: it might just be the reboot. The machine had been up eleven weeks, the update required a restart, and one before/after cannot separate two things that arrived together.

That post also said the experiment could never be rerun:

I cannot rerun the before-case: the old kernel and driver are gone, and even if I rolled them back the machine would have been freshly rebooted.

That was correct, and it stayed correct until an unrelated mistake handed me the missing condition.

How the control appeared

I was setting up an unrelated benchmark and gave llama-server a context window the machine could not hold. It went into memory exhaustion — sshd could accept TCP connections but could not fork a session, so there was no way in. The fix was to pull the power.

The machine came back freshly rebooted and still on the old software. That is precisely the state the August post said was unreachable: the before case with the reboot already spent.

So I measured it, then applied the pending updates, rebooted again, and measured a second time. Both arms sit minutes after a boot. The reboot is held constant. Whatever moves is the software.

What the update actually changed

This one moved only the kernel, which is a stroke of luck the August update did not offer — that one changed the kernel and the driver together, so even a perfect reboot control could not have said which mattered.

The twelve pairs

Three models, four scenarios each, five repetitions, same command both times. All six runs passed the coherence gate 3/3 with zero failures.

Mean +0.33%. Median +0.22%. Range −0.41% to +1.42%. Eight of twelve up.

Why eight-of-twelve is the whole answer

The interesting number is not the mean. It is the count.

In August, twelve measurements out of twelve improved. If software changes nothing, each measurement is a coin flip, and twelve heads in a row happens with probability 0.00024. That is not noise; something systematic moved.

Here, eight of twelve improved. Under the same test that is p = 0.19 — the single most ordinary outcome you can get from twelve coin flips. Four measurements went down, which never happened once in August.

The magnitudes agree. A +0.33% mean sits comfortably inside the run-to-run spread these scenarios already show: ±1.49% on the 35B’s short chat, ±2.10% on the 27B’s code generation. Several of the individual deltas are smaller than the standard deviation of the five repetitions that produced them.

So the two experiments differ in kind, not degree. One is a systematic shift across every measurement. This one is scatter around zero.

The conclusion

The 3.4% was the reboot. Not the update.

Something about a DGX Spark that has been serving models continuously for eleven weeks makes it a few percent slower than the same machine minutes after boot — thermal state, memory fragmentation, accumulated driver or allocator state, something else. This experiment does not identify it. It only shows the software was not responsible, which is what the August post could not say.

For anyone benchmarking a machine like this, the practical form is: uptime is a variable you are measuring whether you meant to or not. If your before and after differ in uptime, you have not measured your change.

Total run time

Half an hour for the measurement that settled a three-week-old question.

That is the point worth taking from this post. The original ambiguity — update or reboot — persisted for three weeks not because resolving it was expensive, but because the control was not available: the machine had to be rebooted without new software, and there was no reason to do that on purpose.

Once an accidental outage supplied it, twelve paired measurements across three models cost about thirty-five minutes. Throughput is cheap to measure. What is expensive is arranging conditions under which a measurement means something.

What this does not tell you

Kernel only. The driver did not move. A future update that changes the driver could genuinely shift throughput, and this result says nothing about that case.

Both arms are warm. The August measurement followed eleven weeks of uptime; both of these follow a boot by minutes. That is what makes the software comparison clean, and it also means “the reboot explains August” is a strong inference from a related condition rather than a direct measurement of it. The honest experiment would run the machine for eleven weeks and measure again without rebooting. That takes eleven weeks.

One update, three models, twelve numbers. A null result on this update is not a general claim that DGX Spark updates do nothing.

Nothing here is about quality. These are throughput measurements only.

A note on how this happened

The control that made this post possible came from an outage I caused, on a machine running production services, at two in the morning. That is not a methodology. It is luck, and the honest framing is that a question this site left open for three weeks got answered by an accident rather than by a plan.

The version worth planning is the one that is still open: let the machine run for months, then measure it, then reboot without changing anything, then measure again. That isolates uptime directly instead of inferring it from the absence of a software effect. It is on the list.

Source data: d6bda83fdac297be9b99e11d7ec3fd69a5a78681d14a44d49a59f7405019082a7a8bef62