Triage MaxText training job logs to identify status and failure modes.
npx skills add https://github.com/AMD-AGI/maxtext-slurm --skill job-log-triage
Please help me install this Agent Skill. Skill: job-log-triage Source: https://github.com/AMD-AGI/maxtext-slurm/tree/main/skills/job-log-triage Command: npx skills add https://github.com/AMD-AGI/maxtext-slurm --skill job-log-triage
Triage and diagnose MaxText training job logs to quickly identify status, failures, and recommended next steps.
Triage a specific job's logs to determine whether it is running, hanging, or failed.
High-intent search queries and answers about installing and using this skill.
Diagnose MaxText training job logs by parsing JOB SUMMARY blocks, completed step lines, and failure signatures from outputs/, ray_logs/, and Slurm summaries to identify stalls, timeouts, NCCL errors, or OOM. It correlates Slurm logs with per-worker logs to validate progress and recommend next steps.
Checking if a Slurm training job is hanging or completed involves locating log files, outputs/, and per-job artifacts to determine progress. By parsing JOB SUMMARY blocks and completed step lines, you can verify whether the job is running, hanging, failed, or successfully completed.
Log triage applies to both RAY=0 and RAY=1 configurations for local and Slurm runs. It locates ray_logs/ and per-job artifacts to analyze training progress, parse step lines, and detect failure modes regardless of the Ray execution mode.
Detect NCCL and OOM timeout errors in training logs by scanning for common failure signatures within the log files. The triage process identifies these specific error patterns and proposes concrete actions to resolve the hardware communication or memory issues.
MaxText job stalls without explicit error messages often indicate silent failures detected by cross-view analysis. Correlating Slurm logs with per-worker logs validates actual training progress and diagnoses stalls by checking completed step lines and JOB SUMMARY blocks.
EnkeZExecute commands and inspect logs on a Ray cluster head node via SSH.
NVIDIAAnalyze PyTorch NCCL flight-recorder dumps to identify collective hangs and suspect ranks.
Supporter09Coordinate distributed model training across Ray clusters with checkpointing and metric reporting.
haozhx23Diagnose NCCL failures on SageMaker HyperPod clusters via events, security groups, and node status.
