How we measure

These rules exist because we broke every one of them once and got burned. Each has a scar behind it.

• Only delivered tokens count. Not the server log's generation counter (it includes speculative tokens that got thrown away), not stream chunks (those are steps, not tokens), not "accepted throughput." Tokens that arrived at a real client — nothing else. The 40.7-vs-27.8 story is why.
• n ≥ 15 runs with a 95% confidence interval — never a median of 3. Per-run variance on this hardware is 15–19%. We once measured the same config at 30.2 and 22.2 on different days; its true value was 25.4 ± 1.9. A median-of-3 is a coin flip wearing a lab coat.
• The restart is the unit of replication. Two server instances of an identical config produced non-overlapping confidence intervals. Any A/B where each arm ran in a single instance is confounded, full stop — this is how a plausible-looking result becomes a wrong default.
• Prefill, decode, and aggregate are reported separately, labeled. They differ by up to two orders of magnitude and vendors mix them on purpose. A number without its label is not a measurement, it's a vibe.
• Both code and prose workloads, always. Speculative decoding can win on one and lose on the other; a benchmark that only ran one workload has measured half the model.
• Temperature 0, fixed prompts, warm-up excluded. And we verify clocks are actually at full speed under load before trusting anything — one node silently stuck at a quarter clock once poisoned a week of numbers.
• Negative controls. When a change "wins," we also measure a change that should do nothing. If the do-nothing change also "wins," the harness is lying, not the config.
• Refuted hypotheses get published. The benchmark entries include what didn't work and what's unfinished. A database of only victories is advertising.