debug-distributed-hang

To debug distributed hangs in SGLang, enable per-rank logging and use a binary-search methodology to locate the first diverge point across ranks during NCCL collectives, then apply fixes to restore progress.

SGLang distributed inference hangs during AllGather or AllReduce when ranks diverge in state during collectives. You can identify the specific deadlock cause by inspecting NCCL logs and analyzing CUDA coredumps.

The best way to find mismatched inputs in model-parallel inference is capturing per-rank logging patterns to diff rank states, then applying a binary-search methodology to pinpoint the first divergence point.

Yes, this diagnosis workflow applies to TP, PP, DP, and EP setups in SGLang. It guides you through using watchdogs, collecting stack traces with py-spy, and inspecting NCCL logs to ensure consistent convergence.

Resolving NCCL deadlocks in SGLang requires collecting stack traces, inspecting NCCL logs, enabling per-rank logging, and checking CUDA coredumps to identify rank divergence and implement the correct fixes.