debug-distributed-hang

Debug a multi-GPU distributed inference hang by attaching py-spy, enabling NCCL debug logs, and configuring per-rank logging to trace stall points, then diff logs to identify and fix the root cause.

Deadlocks in multi-GPU distributed inference setups are caused by size mismatches, non-determinism, conditional branches, and pipeline mismatches. Per-rank logging helps compare GPU states to identify which divergence causes the hang.

Use NCCL logs to locate hang points by enabling debug logging and configuring per-rank outputs from each GPU rank. Diff these logs to trace back to specific inputs or operations causing the distributed training process to stall.

Yes, py-spy works with CUDA to diagnose hanging issues in deep learning inference. Attach py-spy to inspect stalled processes and use CUDA coredumps alongside NCCL logs to locate the exact hang points in multi-GPU operations.

The best way to identify rank divergence causing a distributed GPU hang is per-rank logging. By configuring detailed logs from each GPU rank, you can compare states and trace divergences back to specific inputs or operations.

Your distributed inference process stalls due to size mismatches, non-determinism, conditional branches, or pipeline mismatches across GPU ranks. Analyze CUDA coredumps and diff per-rank logs to find the exact operation causing the deadlock.