debug-distributed-hang

To debug distributed hangs in multi-GPU SGLang deployments, attach py-spy to the scheduler, enable NCCL debug logging, and trigger a CUDA core dump to locate the first diverging rank during collective operations like AllGather and AllReduce.

NCCL collective operations deadlock in distributed inference because ranks diverge in state, causing synchronization failures during AllGather, AllReduce, or Broadcast operations. Identifying the first diverging event requires diffing per-rank logs.

Yes, diagnosing SGLang multi-GPU hangs requires py-spy, NCCL debug logs, and CUDA core dumps. These tools provide deterministic investigation capabilities to pinpoint exact hang locations and validate subsequent fixes.

Locate the first diverging rank when a distributed run hangs by enabling per-rank logging and applying binary-search style techniques. Diffing these rank-specific progress logs isolates the exact state drift causing the deadlock.

Yes, this NCCL hang debugging workflow applies to TP, PP, DP, and EP setups across multi-node clusters. It specifically detects size mismatches, branch divergences, and cascading state drift during collective operations.