resiliency

Recover distributed training from rank hangs by configuring the resiliency extension in Megatron Bridge to enable section-based timeouts, fault-tolerant automatic restarts, and NVRx straggler detection.

Handle Slurm preemption during long-running GPU training by using the resiliency extension's preemption mechanisms to ensure graceful shutdown and state persistence before the job is terminated.

Straggler detection works by monitoring GPU performance individually and relative to other ranks over time to flag slow ranks, which can optionally trigger a training stop to prevent silent performance degradation.

Yes, you can restart training in-process after a failure using the experimental in-process restart and re-run state-machine checks to validate workflows and attribute NaN or spiky-loss issues.

No, you do not use torchrun; you must launch fault-tolerant distributed training using ft_launcher with appropriate environment variables, timeout constraints, and a configured rendezvous endpoint.

The limitation of experimental in-process restart is that it relies on state-machine checks for NaN and spiky-loss attribution, and requires specific ft_launcher environment variables and timeout configurations to execute safely.