gpt-oss ignores its reasoning_effort setting

Both models on this machine ship a reasoning-effort dial. gpt-oss-120b defaults to reasoning_effort: medium, Muse Glimmer to reasoning_strength: high. Both live in the chat template rather than in an API parameter, so nothing in the request names a level and you benchmark whatever the template happens to say.

I set out to publish both curves. One of them turned out to be flat, and proving that it was genuinely flat — rather than that I had broken the experiment — took more work than measuring it.

20,000 questions later: gpt-oss’s dial does nothing, Muse Glimmer’s does a great deal, and the difference between those two sentences is the post.

The two curves

Identical harness, identical questions, 2,800 per arm.

Across its entire effort range gpt-oss moves 0.64 accuracy points — a quarter of the ±2.28 interval — and 0.3% in generation volume. Muse Glimmer moves 3.18 points and 80%.

That is the whole finding. A model asked to think harder that generates one extra token per question is not thinking harder.

Ruling out the boring explanations

A null result is only worth publishing if you have eliminated the ways you could have produced one by accident. There were four.

Did the instruction reach the model? The chat template patch could have silently failed. I asked the server what it actually renders, using its own /tokenize and /detokenize endpoints:

<|start|>system<|message|>You are ChatGPT, a large language model trained by OpenAI.
Knowledge cutoff: 2024-06
Current date: 2026-08-16

Reasoning: high

It arrives.

Did lm-eval send something the server re-templated? If the harness applied the chat template client-side and sent the result as a message, vLLM would wrap it again and the model would read an inner system block with the old setting. It does not: with tokenized_requests=False the harness sends structured messages and the server templates once.

Could the measurement detect an effect at all? Generation volume across the four models on this machine spans 4× — 226 to 2,207 tokens per question. The instrument is not blind.

Was it too early to tell? This is the one I got wrong in public. Three subjects in, I said the null looked real. Then I noticed those three were biology, business and chemistry — recall-heavy subjects where deliberation should help least — and that math, physics and engineering had not run. I flagged it and waited. They ran. Nothing changed.

The positive control I did not plan for

The strongest evidence came from the other model. Muse Glimmer was measured on the same harness, the same benchmark, the same greedy decoding, the same five-shot prompting, the same machine, the same week — and its dial works.

Whatever might have suppressed gpt-oss’s setting would have to leave Muse Glimmer’s alone. Few-shot anchoring and greedy decoding both fail that test.

Its shipped high default is also the worst of its three settings on both axes at once — a finding with its own post.

For contrast, the same table for gpt-oss:

The experiment that closed it

Two explanations survived the sweeps, and both were plausible enough to need killing properly rather than arguing away.

Few-shot anchoring — five worked examples demonstrate a chain-of-thought length, and the model imitates them over a one-line system directive. Greedy decoding — a one-word prompt change perturbs logits, but argmax snaps back to the same trajectory.

So: effort × shots × decoding, 2×2×2, on math (reasoning-heavy) and psychology (fast recall control), 200 questions per cell. The primary readout is tokens per question, because generation length averages over hundreds of long samples and is far more sensitive than a proportion.

Nothing moves anywhere. Largest single deviation 4.7%, mean −0.19%, signs scattered in both directions — noise, not a suppressed effect.

• Not few-shot anchoring. Removing the examples entirely does not wake it up.
• Not greedy decoding. Sampling reveals nothing argmax was hiding.

The ablation script refuses to run against a server that does not render the effort level it was told to expect. That check is the one I wish I had written before the first sweep rather than after.

One thing that did move

Few-shot prompting genuinely does anchor generation length — it just is not what suppresses the effort setting. Independent of effort, five-shot answers are 7.7% shorter on math and 15.4% shorter on psychology than zero-shot.

So the mechanism I suspected is real. It is simply not the culprit, which is the difference between a hypothesis and a finding.

What this means if you run gpt-oss

On vLLM 0.19.0 on this hardware, setting reasoning_effort is not doing anything. If you have been running it at high expecting better answers, you have been paying nothing and receiving nothing — the setting is free because it is inert.

Two things I am not claiming. I have not shown the dial is broken in gpt-oss itself; this is one serving stack at one version, and the same weights under a different runtime may behave differently. And I have not shown effort dials are useless in general — the model in the next column proves the opposite.

The narrow, defensible claim: an effort setting that arrives at the model and changes nothing is indistinguishable from one you never set, and you should verify yours rather than assume it. The check costs one request:

POST /tokenize   {"model": ..., "messages": [{"role":"user","content":"hi"}]}
POST /detokenize {"model": ..., "tokens": [...]}

If the rendered prompt does not contain the level you set, stop there. If it does, measure generation volume at two settings before trusting accuracy numbers — length is where an effect shows first, and where its absence is unmistakable.

Total run time

Six full fourteen-subject sweeps, 16,800 questions, about 22 hours.

The run time is itself a result here. gpt-oss-120b’s three arms took 3.8, 3.9 and 3.8 hours — indistinguishable, because the model generated within 0.3% of the same volume at every effort level. Muse Glimmer’s three took 2.7, 3.7 and 4.5 hours, rising with the setting, because that model actually does more work when asked to.

A model that honours a reasoning-effort setting gets measurably slower as you raise it. That is the cheapest available check, and it disagrees with gpt-oss’s own documentation before any accuracy number is computed.

What this does not settle

One serving stack, one version. vLLM 0.19.0 via nvcr.io/nvidia/vllm:26.04-py3. Worth re-testing on a newer build.

One benchmark shape. MMLU-Pro is multiple choice with a known answer. A model that would think longer on open-ended generation might have nowhere to put the extra effort here — though the ablation’s zero-shot arms argue against that, since removing the answer-format scaffolding changed nothing.

Extraction failures are measured but not compared. 4.5% of gpt-oss’s answers in the logged run failed the extraction regex against 1.2% for Muse Glimmer. The default runs predate per-question logging, so I cannot compare failure rates across effort levels — if high failed extraction more often, some of the flatness could be two effects cancelling. Unlikely at these magnitudes, but unmeasured.

Setup

• NVIDIA GB10, 121 GB unified memory
• vLLM 0.19.0, nvcr.io/nvidia/vllm:26.04-py3, 32k context, --gpu-memory-utilization 0.65
• lm-evaluation-harness 0.4.12, local-chat-completions, 8 concurrent requests
• mmlu_pro_fair with permissive answer extraction and the truncating stop-sequence removed — full scoring method
• Effort set by patching one word in the chat template’s default and serving with --chat-template, since lm-eval cannot pass chat_template_kwargs
• Sweeps: 5-shot CoT, greedy, --limit 200 per subject = 2,800 per arm, three arms per model
• Ablation: 200 questions per cell, temperature=0.7 for the sampled arms
• 20,000 questions total; result files in data/subject-sweep/ and data/effort-ablation/