debug-distributed-hang

Diagnose distributed SGLang hangs by collecting py-spy traces, watchdog behavior, NCCL logs, and CUDA coredumps to locate rank divergence points where collective operations stall.

The best way to find root cause of NCCL collective stalls is enabling per-rank logging to identify state drift, then applying a binary-search workflow to pinpoint the first divergence point and trace inputs to their roots.

Use py-spy traces alongside watchdog behavior and NCCL logs in multi-GPU inference hangs to locate exact stall points, enabling per-rank logging to identify divergence and state drift across GPUs.

Yes, the debugging approach applies in multi-GPU TP, PP, DP, and EP contexts to identify where collectives stall, log per-rank state, and outline practical fixes for safe operational patterns in distributed inference.

Ranks diverge and stall during distributed SGLang runs when collective operations desynchronize, requiring per-rank logging patterns and a binary-search workflow to pinpoint the first divergence point and derive actionable fixes.

Analyze CUDA coredumps alongside py-spy traces and NCCL logs when distributed inference hangs to locate stall points, then enable per-rank logging to identify divergence and state drift across GPUs for actionable fixes.