log-analysis

Diagnoses SLURM training job failures and recommends restart viability per cycle.

320|60|Updated Oct 4, 2024

Search Tags:#llm #root-cause #log-analysis #slurm #nvrx #restart-decision #workload-resilience

Ready to run

npx skills add https://github.com/NVIDIA/nvidia-resiliency-ext --skill log-analysis-nvidia

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: log-analysis
Source: https://github.com/NVIDIA/nvidia-resiliency-ext/tree/main/src/nvidia_resiliency_ext/skills/nvrx-attr/log-analysis
Command: npx skills add https://github.com/NVIDIA/nvidia-resiliency-ext --skill log-analysis-nvidia

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires logsage, langchain-openai, and includes scripts (resource) components.

What problem does it solve?

Analyze SLURM training job logs to attribute failure root causes and guide restart decisions for resilient AI workloads.

Core Features & Use Cases

Per-cycle log analysis and error extraction to map failures to causes.
LLM-assisted classification for complex failure reasons and recommended actions.
CLI and programmatic API access to integrate with training workflows.

Quick Start

Analyze a SLURM job log to obtain per-cycle failure attributions and restart recommendations.

Frequently Asked Questions about log-analysis

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I analyze SLURM job logs to find the root cause of training failures?▼

Analyzing SLURM job logs to find failure root causes involves extracting errors from per-cycle training logs and applying pattern matching to map them to specific causes. This process determines restart viability for resilient AI workloads.

What is the best way to determine if a failed SLURM training job is safe to restart?▼

Determining SLURM job restart viability requires analyzing per-cycle failure attributions to understand the root cause. By extracting errors and applying LLM-assisted classification, you can evaluate whether the underlying issue is safe for a restart.

Can I use an LLM to classify complex failure reasons in SLURM training logs?▼

Yes, you can use an LLM to classify complex failure reasons in SLURM training logs. LLM-assisted classification evaluates extracted error patterns to provide recommended actions and guide restart decisions for failed training jobs.

Do I need a specific Python environment to run NVRxLogAnalyzer for log analysis?▼

Yes, running NVRxLogAnalyzer requires a Python environment with necessary dependencies like logsage and langchain-openai installed. You also need access to the SLURM job logs you intend to analyze for failure root causes.

How do I integrate SLURM log analysis into my existing AI training workflows?▼

You can integrate SLURM log analysis into AI training workflows using CLI or programmatic API access. This allows you to programmatically obtain per-cycle failure attributions and restart recommendations during the training process.