debug-distributed-hang

Debug multi-GPU distributed inference hangs in SGLang by tracing per-rank state, watchdog traces, and NCCL logs to identify divergence points and stalled collectives. You can then apply a structured workflow to binary-search the root cause and validate fixes.

All ranks stall during NCCL collective operations when there is a divergence in execution state across GPUs or misrouted communications in TP/PP/DP/EP configurations. Per-rank logging and NCCL debug logs help surface the specific rank where the hang originates.

Use py-spy alongside CUDA debugging tools and NCCL debug logs to profile and dump the state of stalled ranks during a multi-GPU hang. This combination surfaces exactly where the collective operation freezes and helps trace the root cause of the divergence.

You need CUDA debugging tools, py-spy, and NCCL logs to reproduce and validate fixes for multi-GPU SGLang inference stalls. These tools are required to trace per-rank state, analyze core dumps, and confirm that the applied resolution prevents future hangs.

The best way to locate rank divergence in TP/PP/DP/EP configurations is to enable per-rank logging and analyze watchdog traces to find the first rank that deviates. This structured detection workflow isolates the exact point of misrouted communications or stalled collectives.