skills.rest
NVIDIANVIDIAOfficial·4 Agent Skills Included

nvidia-resiliency-ext

Diagnose distributed training failures and automate restart decisions

Analyzes SLURM job logs and NCCL flight-recorder dumps to pinpoint why distributed training jobs failed and which ranks caused hangs. Eliminates hours of manual log digging and guesswork when large GPU clusters stall or crash. Runs automated fault-injection benchmarks that score root-cause accuracy and drive continuous improvement of failure attribution.
320|60
↗
npx skills add NVIDIA/nvidia-resiliency-ext --all -g -y
Available:

Gives your AI agent the build, test, and architecture context for this resiliency library so it can navigate modules like fault tolerance, checkpointing, and attribution when executing the bundled skills.

All Skills in This Repository (4)

Pure Emerald Level Indicators
📦 In Repo
NVIDIANVIDIA

nvrx-attr

Coordinate fault-attribution workflows across SLURM-based PyTorch Megatron-LM training jobs.

Official
Advanced
📦 In Repo
NVIDIANVIDIA

log-analysis

Diagnoses SLURM training job failures and recommends restart viability per cycle.

Official
Advanced
📦 In Repo
NVIDIANVIDIA

fault-injection-loop

Submit prioritized fault-injection experiments in pairs via sbatch on SLURM clusters.

Official
Advanced
📦 In Repo
NVIDIANVIDIA

fr-analysis

Analyze PyTorch NCCL flight-recorder dumps to identify collective hangs and suspect ranks.

Official
Advanced

Frequently Asked Questions

FAQPage Schema
How to install nvidia-resiliency-ext?▼

Run `npx skills add NVIDIA/nvidia-resiliency-ext --all -g -y` in your terminal to install all skills in this suite globally.

How to find why a SLURM training job failed?▼

Use the log-analysis skill, which reads your job log, extracts failure evidence, and returns a structured RESTART or STOP decision with the root cause.

How to find which GPU rank caused an NCCL hang?▼

The fr-analysis skill parses PyTorch flight-recorder dumps, detects where collectives diverged, and reports the exact suspect ranks responsible for the hang.

Does log analysis require an LLM API key?▼

An LLM_API_KEY enables richer explanations, but you can run deterministic-only analysis with the --disable-l1 flag and no key.

What do I need to run the fault-injection benchmark?▼

You need SLURM cluster access with sbatch, a configured user.env file, and a Megatron-LM workload, since the loop is validated only on Megatron-LM training jobs.

Related Repositories in Software Engineering

View All in Software Engineering→
openclawopenclaw
388.4k

openclaw

Run a personal AI assistant across your devices and chat apps

Official📦 120 Skills
obraobra
280.0k

superpowers

Gives coding agents a disciplined workflow from idea to merged code

Community📦 15 Skills
facebookfacebook
248.4k

react

AI agent skills for building, testing, and porting React core

Official📦 14 Skills