Why your model hangs forever "loading weights" on GB10

Field note · measured on an 8× GB10 cluster

You start the server, it prints something like loading weights..., and then — nothing. No error, no crash, no traceback. Just a process pinned at 100% of one core, or sitting dead quiet, while minutes tick by. On a big model you convince yourself it's "just slow." Twenty minutes later you're still staring at the same line.

Here's the thing that makes this one maddening on Grace-Blackwell: three completely different root causes all look identical from the outside. Same frozen log line, same non-error. If you don't know which of the three you're in, you'll spend a day fixing the wrong one. So the entire skill here is telling them apart — that's what this note is for.

It's almost never "the model is big"

The first instinct — "128GB of weights just takes a while" — is usually wrong. A healthy load on this hardware is far faster than people expect once the path is clean. We've taken a load that sat for around seven and a half minutes down to roughly 35 seconds without changing the model or the storage at all — just by fixing which path the bytes actually travel. If your load is measured in many minutes, that's a signal something is wrong, not a law of physics.

The three suspects, and how they differ

When a GB10 load hangs, it's overwhelmingly one of these three. They feel the same; they are not the same:

• Suspect 1 — the bytes are crawling to the GPU. There's a specific way weights can end up being copied to the GPU that quietly runs at a tiny fraction of the bandwidth you'd expect — think hundreds of MB/s where you should be seeing gigabytes. The load "works," it just takes forever. The tell is in how the memory is being read before it's copied, and it's a one-line-class fix once you see it — but only if you know to look there and not at your disk or your model.
• Suspect 2 — one CPU core is doing all the work. The classic signature is a single core pinned at 100% while the GPUs sit idle and the other 19 cores do nothing. That's a serialized, per-chunk loader doing sequentially what it could do in parallel. Storage isn't the bottleneck and sharding won't save you — the loader itself is the wall.
• Suspect 3 — it's not loading at all, it's waiting. On multi-node, a "loading" hang is frequently not loading — it's rank 0 sitting at a barrier waiting for peers that never check in, because one environment value that tells the ranks how to find each other is unset or wrong. Nothing is slow; everything is stuck. This one looks exactly like the other two and has nothing to do with weights.

That triage is the part that saves you the day. The actual fix for each — where exactly to change the copy path, how to parallelize the loader safely, and which environment value has to be set to what on multi-node — is specific to your framework version, your model's shard layout, and your cluster topology. Get the diagnosis right and the fix is quick; guess the diagnosis and you'll "fix" a problem you don't have.

Free to read and share. Numbers are from hardware we've measured directly; your setup will differ with framework version, model layout, and topology — measure your own.