debug-distributed-hang

Multi-GPU SGLang inference jobs hang during NCCL AllGather operations when GPU ranks diverge during collective communication. This Skill diagnoses the freeze by locating blocked collective operations and identifying the first divergent rank state using watchdog or py-spy traces.

To debug an NCCL collective hang in distributed tensor parallel inference, you need watchdog or py-spy traces and NCCL diagnostics to localize the blocked operation. This Skill guides you through adding per-rank structured logging to compare event counts and locate the state divergence.

Finding the first divergent rank state in a frozen pipeline parallel inference job involves adding structured per-rank logs and comparing event hashes. This Skill traces divergent inputs backward to address floating-point nondeterminism, random state differences, and pipeline send or receive mismatches.

Yes, you can use optional CUDA coredumps alongside NCCL logging and per-rank structured logging to diagnose rank divergence in multi-GPU inference. This Skill uses these diagnostics to identify the blocked code path and verify the fix across repeated validation runs.

Yes, this distributed debugging approach works with SGLang data and expert parallelism runs that freeze or time out. It applies to tensor, pipeline, data, and expert parallelism configurations experiencing rank divergence during point-to-point or collective communication.

To fix floating-point nondeterminism causing distributed inference hangs, trace divergent inputs backward to locate the first rank-state divergence. This Skill helps address floating-point nondeterminism, random state differences, and conditional branch divergence, followed by repeated validation of fixes.