Verified:GitHub

nemo-mbridge-resiliency

Configures Megatron Bridge fault tolerance, straggler detection, and restart for distributed GPU training.

Updated Apr 16, 2026

Search Tags:#checkpointing #preemption #fault-tolerance #resiliency #megatron-bridge #straggler-detection #in-process-restart

Ready to run

npx skills add https://github.com/sayalinvidia/sayali-skills-test --skill nemo-mbridge-resiliency

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: nemo-mbridge-resiliency
Source: https://github.com/sayalinvidia/sayali-skills-test/tree/main/skills/Megatron-Bridge/nemo-mbridge-resiliency
Command: npx skills add https://github.com/sayalinvidia/sayali-skills-test --skill nemo-mbridge-resiliency

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Megatron Bridge resilience features address training hangs, faults, and straggler issues by enabling fault tolerance, straggler detection, and restart mechanisms across distributed GPU training workflows.

Core Features & Use Cases

Fault tolerance for Slurm and NeMo-Run integration with nvidia-resiliency-ext
NVRx straggler detection and configurable restart options
In-process restart (experimental) for in-process resilience
Preemption support with safe checkpointing and graceful shutdown
Experimental re-run state machine with NaN/spiky loss attribution
Async and local checkpointing for faster recovery and reduced downtime

Quick Start

Configure the resiliency features according to the Megatron Bridge documentation and validate via tests and examples.

Frequently Asked Questions about nemo-mbridge-resiliency

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I add fault tolerance to distributed GPU training workloads?▼

Fault tolerance for distributed GPU training is configured by integrating nvidia-resiliency-ext with Megatron Bridge and NeMo-Run. This enables robust recovery mechanisms to prevent training hangs and faults across large-scale cluster environments.

Can I detect and handle stragglers in Megatron Bridge training?▼

Yes, straggler detection in Megatron Bridge is handled using NVRx to identify slow ranks causing delays. Configurable restart options allow the system to automatically recover from these stragglers and maintain training throughput.

Does preemption support include safe checkpointing for graceful shutdown?▼

Preemption support in Megatron Bridge includes safe checkpointing and graceful shutdown capabilities. This ensures that distributed training workloads can save their state and stop cleanly when preempted by cluster schedulers like Slurm.

What is the recommended fault tolerance path for NeMo-Run integration?▼

The recommended fault tolerance path for NeMo-Run integration requires specific launcher configurations and utilizes nvidia-resiliency-ext. It targets large distributed GPU training workloads to ensure reliability across Slurm or similar cluster environments.

How does in-process restart work for training resilience?▼

In-process restart is an experimental resilience feature in Megatron Bridge that allows training to recover from faults without fully exiting the process. This reduces downtime and speeds up recovery during distributed GPU training.

Are there experimental features for NaN loss attribution in training?▼

Yes, an experimental re-run state machine provides NaN and spiky loss attribution during training. Combined with async and local checkpointing, this reduces downtime and improves recovery speed for distributed workloads.