debug-distributed-hang

Debug distributed SGLang hangs by using py-spy, watchdog, and NCCL logs to identify stall locations across TP/PP/DP/EP ranks. Analyze per-rank logs to find divergence points and apply binary-search workflows to validate fixes.

NCCL collectives like AllGather or AllReduce stall when ranks diverge across GPUs. Use watchdog-triggered timeouts and CUDA coredumps to detect the hang, then diff per-rank logs to pinpoint the exact divergence point causing the timeout.

Locate the first rank divergence by enabling per-rank logging and diffing the outputs across GPUs. Apply the binary-search root cause workflow to systematically narrow down the initial point of failure during distributed runs.

Common causes of distributed inference hangs include tensor size mismatches, branch divergence, and pipeline issues across TP and PP ranks. Step-by-step verification through repeated runs confirms whether applied fixes resolve the divergence.

You can use py-spy and CUDA coredumps alongside NCCL debugging to diagnose SGLang watchdog timeouts. These tools capture the exact hang location and state of multi-GPU processes when collectives stall during distributed execution.