The traps our engineers hit bringing up multi-node Grace-Blackwell — and the measured fixes. Everything here is real numbers from our own clusters. If it saves you a week, and you'd rather hand the next one off, that's what the day-rate is for.
Kimi K2-class ~1T MoE at 26–28 tok/s on 8 nodes. Qwen3.6-Next at 45.2 on one. A 753B and a ~400B MoE in between — every number delivered-tokens, n≥15 or median protocol, with the tuning journey described.
free toolModel size, quant, node count in — fit verdict and expected tokens/sec out. Calibrated against our measured cluster results, including the multi-node overhead spec-sheet math ignores.
index · start hereEvery trap we've root-caused on real multi-node Grace-Blackwell hardware, in one list: the symptom you'll see, and where the problem actually lives. Fabric, loading, unified memory, spec decode, silent wrong-output bugs, and the monitoring tools that lie.
networking · multi-node~94 on a 100G fabric is the win condition, not a shortfall. When to stop debugging, why the rails don't add up, and the two misconfigurations that actually explain a flow stuck down at 13 Gbps.
vLLM · loadingThree different root causes that all look identical from the outside — a crawling copy path to the GPU, a single-core serialized loader, and a multi-node barrier that's waiting, not loading. How to tell them apart before you waste a day on the wrong one.
performance · prefillDecode fine, fabric at line rate — and prefill 4.4× more expensive than it had to be, because the collective library picked a latency protocol for 96 MB messages. One line: 699 → 1,520 tok/s, TTFT 14.6 s → 6.7 s.
spec-decode · cuda graphsAcceptance healthy, draft cheap — and spec-on ≈ spec-off, because enabling speculation silently disabled all CUDA-graph capture. Restoring the right partial-capture mode recovered +58%. The trap nobody documents.
networking · multi-nodeTP=2 works, TP=4 fails or gets slower — because tensor parallelism puts the network on the critical path of every token. When pipeline parallelism wins instead, and how to think about topology on a 100 Gbps switch.
networking · RDMAUnified memory means no zero-copy NIC-to-GPU path, so NCCL silently hangs or crawls on multi-node. Why the wire tests fast while collectives crawl — and the config that recovers the bandwidth.
quantization · correctnessOn sm_121, FP4 kernels pass their own support check then emit "!!!!!" or all-NaN logits with no error. Why nvfp4 is a trap on Spark by default, and how to tell a kernel bug from a real model problem.
performance · long contextPrompt processing falls off hard as context grows — but a lot of "long context is unusable" is fixable config, not attention physics. How to separate the two and keep long context usable.
operations · air-gapUpdates, model loading, and telemetry all assume internet. Why /etc/hosts is not an air gap, the offline-update question NVIDIA's docs skip, and how to prove the seal at the wire.
memory · operationsCPU and GPU share one pool, so MemAvailable misleads, "used" RAM is often reclaimable cache, and a too-high GPU fraction wedges the server. How to account for 128 GB you can actually trust.
measurement · benchmarkingThree throughput numbers from the same server in the same window — and only one of them is real for your users. Proposed vs. accepted vs. delivered tokens, steps/sec masquerading as tokens/sec, and the restart variance that confounds most A/B tests.
The notes are free. If you'd rather not spend weeks finding this out yourself, we reproduce your exact setup on real GB10 hardware and hand back the fix.
Book a 1-day diagnostic →A short note when we publish a new trap, benchmark entry, or tuning result. No marketing, just the numbers — unsubscribe is one click.
Subscribe by email →We reply from a real mailbox and add you by hand. No tracker, no third-party form.