Speculative decoding made the model slower. Here's the trap — and the +58%.
Field note · measured on GB10, large sparse-MoE model
On paper, speculative decoding is close to free speed: a small draft model guesses several tokens, the big model verifies them in one pass, and every accepted guess is a token you didn't pay full price for. We measured healthy acceptance. The draft model was fine. And the end-to-end result was net neutral — sometimes slower than no speculation at all.
If that sentence describes your cluster, you're probably about to conclude "spec-decode doesn't work on this hardware." That conclusion is common. That conclusion is wrong, and the reason it's wrong is a silent interaction most stacks never surface.
The silent trade you didn't agree to
Modern inference engines rely on CUDA-graph capture to amortize kernel-launch overhead — instead of the CPU launching thousands of tiny GPU operations per token, whole stretches get replayed as one pre-recorded graph. On a CPU-modest, GPU-modest platform like GB10, that amortization is worth a lot: launch overhead that disappears in the noise on a datacenter box is a first-order cost here.
Here's the trap: on the stack we tested, enabling speculative decoding silently disabled all graph capture. No warning, no log line at default verbosity — spec went on, capture went off, and the launch overhead it reintroduced ate the entire speculative win. Two features, each individually good, quietly cancelling each other. Every benchmark says "spec ≈ no gain," and the real story — you traded one optimization for the other — appears nowhere.
The signature: acceptance length looks healthy, the draft model is cheap, and yet spec-on ≈ spec-off end to end. Before concluding speculation is broken, go find out what capture mode your engine was actually running in with spec enabled — not what you configured; what it resolved to at startup.
The way out — and the number
Full graph capture and speculation genuinely do conflict in some engine versions — that part is real. But it's not all-or-nothing: there are intermediate capture modes that coexist with speculative decoding, capturing what can be captured and leaving the rest eager. Moving from "silently no capture" to the right partial-capture mode recovered +58% decode throughput on our workload — with speculation still on, verified across restarts with confidence intervals (because on this platform, restart variance will happily fake a win that size if you let it).
Which capture mode coexists with speculation, on which engine version, with which draft configuration — and whether your verify pass is itself capturable — is exactly the kind of three-way version dance that eats a week. It also interacts with how many tokens you draft: we've measured configurations where drafting more tokens made everything slower because acceptance, not depth, was the ceiling.
Spec-decode not paying on your cluster?
Book a one-day diagnostic with Anantus. We determine what your engine is actually doing with capture and speculation together — measured on real GB10 hardware, with the before/after and confidence intervals to prove which configuration wins for your workload.
Book a 1-day diagnostic →Free to read and share. Numbers are from hardware we've measured directly; your engine version and draft setup will differ — measure your own.