nemo-mbridge-resiliency

Fault tolerance for distributed GPU training is configured by integrating nvidia-resiliency-ext with Megatron Bridge and NeMo-Run. This enables robust recovery mechanisms to prevent training hangs and faults across large-scale cluster environments.

Yes, straggler detection in Megatron Bridge is handled using NVRx to identify slow ranks causing delays. Configurable restart options allow the system to automatically recover from these stragglers and maintain training throughput.

Preemption support in Megatron Bridge includes safe checkpointing and graceful shutdown capabilities. This ensures that distributed training workloads can save their state and stop cleanly when preempted by cluster schedulers like Slurm.

The recommended fault tolerance path for NeMo-Run integration requires specific launcher configurations and utilizes nvidia-resiliency-ext. It targets large distributed GPU training workloads to ensure reliability across Slurm or similar cluster environments.

In-process restart is an experimental resilience feature in Megatron Bridge that allows training to recover from faults without fully exiting the process. This reduces downtime and speeds up recovery during distributed GPU training.

Yes, an experimental re-run state machine provides NaN and spiky loss attribution during training. Combined with async and local checkpointing, this reduces downtime and improves recovery speed for distributed workloads.