nvrx-attr

Fault attribution for distributed ML training is automated by orchestrating log-analysis, FR-analysis, and fault-injection benchmarking workflows across SLURM-based jobs. This coordination accelerates root-cause isolation and provides a unified scoring report.

Yes, you can orchestrate fault-injection benchmarking with log analysis for PyTorch Megatron-LM workloads. The workflow coordinates fault-injection experiments, collects artifacts, and reuses per-skill entry points to benchmark attribution quality.

The best way to coordinate log and FR analysis for root-cause isolation is by using a central orchestration workflow. It weaves together log-analysis and FR-analysis sub-skills to accelerate automated scoring across distributed ML training jobs.

Yes, you need an active LLM_API_KEY or equivalent credentials to run fault-injection attribution workflows. The environment also requires Python 3.10+, the nvidia-resiliency-ext package, logsage, and langchain-openai.

Yes, the fault-attribution orchestration workflow supports modular usage of individual analysis sub-skills. It reuses per-skill entry points like NVRxLogAnalyzer and CollectiveAnalyzer to evaluate attribution quality on SLURM workloads.

Limitations for end-to-end fault attribution include strict dependencies on SLURM-based scheduling and PyTorch Megatron-LM workloads. It also requires specific Python 3.10+ packages and valid LLM credentials to load on-demand resources successfully.