Integrate nvidia-resiliency-ext fault tolerance into Megatron Bridge training workflows.
npx skills add https://github.com/NVIDIA/skills --skill resiliency
Please help me install this Agent Skill. Skill: resiliency Source: https://github.com/NVIDIA/skills/tree/main/skills/Megatron-Bridge/resiliency Command: npx skills add https://github.com/NVIDIA/skills --skill resiliency
Megatron Bridge resiliency features provide fault tolerance, straggler detection, in-process restart, preemption, and a re-run state machine to improve stability of large-scale training workflows. Use when you need hang detection, automatic restart, or checkpoint recovery in distributed training.
Enable resiliency features by turning on the FT, NVRx straggler, IPR, and checkpointing options in Megatron Bridge.
High-intent search queries and answers about installing and using this skill.
Fault tolerance for distributed ML training is added by integrating nvidia-resiliency-ext into Megatron Bridge, enabling automatic recovery from failures during long runs. Turn on the FT options in Megatron Bridge to start.
Straggler detection in multi-node training identifies slow nodes that hinder resource utilization, allowing graceful handling to maximize efficiency. Enable the NVRx straggler option in Megatron Bridge to activate this monitoring.
In-process restart allows rapid recovery for a hung distributed training job without full job relaunch. Enable the experimental IPR and re-run state machine options in Megatron Bridge to trigger this quick recovery.
Downtime during cluster preemption and checkpointing is minimized using preemption support and non-blocking checkpointing. Enable async checkpointing and local checkpointing in Megatron Bridge to accelerate recovery and reduce progress loss.
Megatron Bridge supports hang detection for large-scale model training by integrating nvidia-resiliency-ext features. Use it when you need automatic restart or checkpoint recovery in distributed training workflows.
Enable resiliency features when your large-scale training workflow requires hang detection, automatic restart, straggler handling, or checkpoint recovery across distributed nodes. Turn on the FT, NVRx straggler, IPR, and checkpointing options in Megatron Bridge.
sayalinvidiaConfigures Megatron Bridge fault tolerance, straggler detection, and restart for distributed GPU training.
yo-stevenImplement fault-tolerant restarts and straggler detection for Megatron Bridge training.
achmfConfigure the Sentry Cocoa SDK for Apple app error monitoring and tracing.
delorenjDiagnose and recover corrupted git state, orphaned worktrees, and inconsistent branch references.
