fr-analysis

You can identify ranks causing an NCCL hang by parsing PyTorch flight-recorder dumps to detect collective operation divergence across process groups and isolating missing ranks as root-cause suspects.

To enable PyTorch NCCL flight-recorder dumps for hang detection, set the TORCH_NCCL_TRACE_BUFFER_SIZE environment variable greater than zero before running your distributed training job to capture the required trace data.

Yes, you can generate a narrative summary of NCCL collective hangs from flight-recorder dumps by running the attribution script with the optional LLM analysis pass enabled via the --llm-analyze flag.

A collective analyzer finds the wavefront boundary in NCCL timeout dumps by grouping collective records by process group and sequence across ranks, identifying where operations diverge and isolating missing ranks as suspects.

The best way to debug distributed PyTorch training jobs hanging on NCCL timeouts is analyzing flight-recorder dumps to identify the wavefront process group where collectives diverged and isolating the responsible missing ranks.