debug-distributed

To fix distributed training hangs and NCCL errors, use rank-aware diagnostics to verify group membership and synchronization across all ranks. You can isolate the failing operation using a minimal reproduction workflow and apply environment variable configurations to resolve communication failures.

FSDP runs out of memory across different ranks due to inconsistent tensor shapes or synchronization issues. You can diagnose these OOM errors by applying rank-aware checks to verify group membership and tensor shapes, then creating a minimal reproduction workflow to isolate the failing operation.

The best way to debug inconsistent results in distributed training is to use rank-aware diagnostics to verify synchronization across all ranks. You should create a small, reproducible example to isolate the failing operation and identify where tensor shapes or group membership diverge.

You create a minimal reproduction workflow for distributed training failures by building a small, reproducible example that isolates the failing operation. This skill guides you through the debugging steps and rank-aware checks needed to systematically troubleshoot issues like hangs, OOM, and NCCL errors.

To troubleshoot NCCL communication errors, you need environment variable configurations and debugging tools that collect actionable signals. This skill provides environment and tooling guidance to help you capture the necessary diagnostic information across ranks to resolve communication failures.