fault-injection-loop
OfficialAutomate closed-loop fault-injection benchmarking.
AuthorNVIDIA
Version1.0.0
Installs0
System Documentation
What problem does it solve?
Closed-loop fault-injection experimentation is automated end-to-end, enabling rapid benchmarking of resilience workflows for large-scale ML training by coordinating a prioritized pool of experiments and submitting them two at a time to bound filesystem load.
Core Features & Use Cases
- Closed-loop fault pool design with prioritized (fault_type, rank, iter, nodes) entries and automated submission.
- Deterministic batching: two simultaneous jobs per round with SBATCH and waiting until both finish.
- Automatic post-run analysis: log-analysis and FR-analysis, then attribution scoring and gap aggregation to guide improvements.
Quick Start
Configure your environment and run the batch workflow with bash scripts/prepare_node_alloc.sh to start submitting fault-injection experiments.
Dependency Matrix
Required Modules
None requiredComponents
scripts
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: fault-injection-loop Download link: https://github.com/NVIDIA/nvidia-resiliency-ext/archive/main.zip#fault-injection-loop Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 471,000+ vetted skills library on demand.