fault-injection-loop

Closed-loop fault-injection benchmarking is automated by identifying prioritized experiments and submitting two jobs at a time via sbatch to bound filesystem load while testing large-scale ML training resilience.

To bound filesystem load during fault-injection experiments, the workflow submits exactly two simultaneous jobs per round using sbatch and waits until both finish before starting the next batch.

Yes, this closed-loop fault-injection workflow is compatible with Megatron-LM workloads, automatically running post-run log-analysis and FR-analysis on experiments to score attribution against ground truth.

Yes, you need logsage and an LLM_API_KEY to run attribution scoring, because the workflow relies on nvidia-resiliency-ext installed tooling to perform post-run log-analysis and FR-analysis.

To iteratively improve fault attribution modules, the workflow automatically runs log-analysis and FR-analysis after each batch completes, scores attribution against ground truth, and aggregates gaps to guide module improvements.

Submitting only two fault-injection jobs at a time via sbatch bounds filesystem load but limits throughput, requiring the workflow to wait until both jobs in a batch finish before proceeding to the next prioritized experiments.