Methodology

Every number on this site is produced by one harness — about 350 lines of dependency-free Python — and it is the same file on every machine. No per-machine tweaks, because a benchmark you edit per machine is not a benchmark.

What is measured

The harness talks to an OpenAI-compatible /v1/chat/completions endpoint and times the response stream from the client side. Measuring client-side is deliberate: llama.cpp, ollama, vLLM and MLX all report internal statistics differently, and some do not report them at all. Timing the stream is the only method that produces numbers comparable across all of them.

• Time to first token (TTFT) — wall-clock from request dispatch to the first content token arriving. This is what makes a model feel responsive or sluggish.
• Decode throughput — tokens per second during generation, measured across the window from the first token to the last. The first token is excluded from the count because it is produced at the start of that window, not during it.
• Prefill throughput — prompt tokens divided by TTFT. This is the number that collapses on memory-bandwidth-limited machines once the context gets long.

How runs are conducted

• Every scenario runs one warmup pass that is discarded, then five measured repetitions. The warmup exists to absorb weight loading and cache effects that would otherwise land entirely on the first measurement.
• Reported values are medians, with standard deviation shown alongside. A median resists the single slow run that thermal throttling or a background process will eventually produce.
• Decoding is greedy (temperature 0) so output length stays stable between repetitions. Throughput compared across runs of different lengths is not a comparison.
• Prompts are built from a fixed filler string at target token counts, so every machine prefills the identical input.

The four scenarios

Scenarios are frozen. Adding one is fine; editing an existing one would silently invalidate every previously published comparison, so it does not happen.

How quality is scored

Speed is only half the question. A model that answers twice as fast and gets it wrong is not faster at anything useful. Quality is measured with a second harness against MMLU-Pro, a public set of 12,032 graduate-level multiple-choice questions spanning fourteen subjects, with up to ten options each rather than the usual four.

No model grades another model here, and I do not grade them either. That is worth stating plainly, because LLM-as-judge scoring has become common enough that readers reasonably assume it. Every question in MMLU-Pro ships with its correct answer as a field in the dataset. Scoring is a string comparison against that answer sheet.

Concretely, one real question and how it is scored:

Question 2804  (biology)
  Which of the following would most likely provide
  examples of mitotic cell divisions?

    A. cross section of muscle tissue
    B. longitudinal section of a shoot tip
    C. longitudinal section of a leaf vein
    ...                      (eight options in this case)

Answer sheet     "answer": "B"

Model writes     "...the shoot tip contains meristematic
                  tissue, which is actively dividing, so
                  the answer is B."

Extracted        B
Scored           correct

The extraction step is a regular expression, not a judgement call:

(?i)(?:answer|correct choice|correct option|best answer)\s*(?:is|:)\s*\**\s*\(?([A-J])\)?

It takes the last match in the response, not the first. That detail matters for reasoning models: one that weighs option B in the middle of its working and concludes D is scored on D, which is what a human marker would do.

• Five-shot, greedy. Each question is preceded by five worked examples from the dataset's own validation split, and decoding is greedy — lm-evaluation-harness sends temperature: 0 and seed: 1234 on every request. Greedy is not reproducible here, and this page used to claim it was. Two runs of the same 200 law questions — same weights, same serving command, same cap — produced zero byte-identical answers out of 200 on either Qwen model, diverging after a median of 1,018 characters on the 35B-A3B and 366 on the 27B. The 35B-A3B scored 64.5 on one run and 67.5 on the other. Under continuous batching the composition of a batch varies with timing, which changes the order of floating-point reductions, which occasionally changes an argmax; over a 10,000-character answer that compounds. Expect a few points of run-to-run movement in any single subject score on this site.
• 200 questions per subject, 2,800 per model. Not a random sample — the first 200 of each subject in dataset order, so every model is asked the identical questions and any rerun is directly comparable.
• Only the letter is compared. Reasoning, working and prose are not assessed at all. The model can arrive at B by any route or none.

What this scoring gets wrong

• Format compliance is counted as knowledge. A model that knows the answer but never phrases it in a way the regex recognises is marked wrong. This is the single biggest weakness of the method, and it punishes chatty reasoning models hardest.
• The stock harness makes that far worse, which is why the task here is a modified fork. Off the shelf it stops generation at the string Question: and caps the response short — together those decapitate a reasoning model mid-thought and score it near zero. The fork removes the stop sequence and raises the cap to 3,072 tokens. There is a whole post about discovering that.
• Small gaps are not real. At 200 questions a single subject carries roughly ±2 points of sampling error, and a full 2,800-question run about ±0.8. Two models a point apart are tied, and this site says so rather than ranking them.
• The questions are public. Any of these models may have encountered them during training, and there is no way to check from the outside. These scores measure standing against a shared yardstick, not absolute capability — which is exactly why the comparison is run on identical questions rather than cited from model cards.

Results land in data/subject-sweep/ as JSON, one file per model, written incrementally after every subject so an interrupted run keeps everything measured up to that point. As with the speed numbers, the tables are rendered from those files at build time.

What the speed numbers are not

Single-stream benchmarks. They describe one person using one model interactively — the actual local-LLM use case — and they are a floor rather than a ceiling for anything serving more than one request at a time. That gap is now measured rather than guessed at: on this machine eight simultaneous requests produce 2.9× the total tokens per second, while each individual request runs at 36% of its solo speed. Read the published figures as the single-user number and that post for the batch one. They also reflect specific driver, backend and model versions, all recorded in each result file. A rerun six months later on the same hardware can legitimately produce different numbers, which is why the backend version is published alongside every result.

Reproducing them

# On the machine under test, with a server already running:
python3 bench/runner.py \
  --url http://localhost:11434 \
  --model llama3.1:8b \
  --label my-machine \
  --quant Q4_K_M

The result lands in data/runs/ as JSON. Those files are the primary record — the tables on this site are rendered from them at build time, so a published number and its source file cannot disagree, and no figure is ever retyped by hand.

The repository is currently private, so that command is not something you can run today. The description above is deliberately complete enough to reimplement: client-side stream timing against an OpenAI-compatible endpoint, one discarded warmup, five measured repetitions, medians with standard deviation, greedy decoding, and the four fixed prompt sizes in the table. If you rebuild it and get materially different numbers on comparable hardware, I would genuinely like to know.