skills.rest
zjunlp
zjunlpOFFICIAL
Explore all skills by this author →
Website𝕏@ChenHuajunGitHub

training-check

Monitors WandB training metrics periodically to detect NaN, divergence, and stalled runs.

75|7|Updated May 2, 2026
Search Tags:#health check#gpu#wandb#llm training#training monitoring#loss divergence#nan detection
One-click install
npx skills add https://github.com/zjunlp/Mechanist --skill training-check-zjunlp
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: training-check
Source: https://github.com/zjunlp/Mechanist/tree/main/skills/training-check
Command: npx skills add https://github.com/zjunlp/Mechanist --skill training-check-zjunlp

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires wandb.

What problem does it solve? Long training runs can silently waste GPU hours when loss diverges, NaN values appear, or metrics plateau, and these problems are often only discovered after training finishes. This Skill performs periodic automated health checks on running training jobs so failures are caught early. ## Core Features & Use Cases - WandB Metric Monitoring: Reads loss trends, eval metrics, gradient norms, and learning rate schedules from the WandB API, with SSH log-file fallback when WandB is unreachable. - Judgment & Escalation: Classifies signals as clearly bad, clearly fine, or ambiguous, escalating only ambiguous cases to an external LLM reviewer for a STOP/CONTINUE/WAIT verdict. - Adaptive Scheduling: Runs via CronCreate with intervals that grow from 10 to 60 minutes when training stays healthy and reset after anomalies. - Use Case: After launching a multi-day LLM fine-tuning run, set up this Skill to check the WandB run every 10 minutes, automatically stopping the job and saving evidence if the loss diverges. ## Quick Start Set up a recurring check for my WandB run at entity/project/run_id and alert me if the training loss starts diverging.

Frequently Asked Questions about training-check

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I monitor WandB training metrics automatically?▼

Use the WandB API to fetch run history and check loss trends, eval metrics, NaN values, and gradient norms on a recurring schedule. This Skill automates that via CronCreate, starting at 10-minute intervals and increasing up to 60 minutes when training stays healthy.

How to detect NaN or loss divergence during model training?▼

Check the loss history from WandB for NaN or Inf values, sudden spikes over 10x normal variance, and sustained increases across multiple checkpoints. Clearly bad signals trigger an immediate stop; ambiguous trends are escalated to an external LLM reviewer.

What happens if WandB is unreachable during a training check?▼

The Skill falls back to reading the training log directly over SSH by tailing the log file on the remote server. If both WandB and logs are unreachable, it reports the connectivity issue and retries at the next interval rather than assuming training is broken.

Does training-check monitor GPU utilization and process health?▼

No, it checks training quality only, such as loss trends and metric improvement. Process health like session liveness and GPU activity should be handled by a separate launcher or monitoring workflow, and the two are designed to run together.

When should I stop a training run based on metrics?▼

Stop when loss contains NaN or Inf, diverges for multiple consecutive steps, or eval metrics fall significantly below baseline. Avoid stopping on a single noisy spike; judge trends over multiple checkpoints and consult the external reviewer for ambiguous cases.

Related Skills

Explore More Software Engineering Skills→
OpenLAIROpenLAIR
1.1k

aris-training-check

Monitors WandB training metrics periodically to detect NaN, divergence, and idle GPUs.

Official
Intermediate
zhuyingqinzhuyingqin
1

training-check

Detect training divergence and quality regressions from WandB metrics.

Community
Intermediate
TabithaFannyTabithaFanny

training-check

Monitor WandB training metrics for NaN values, loss divergence, and idle GPUs.

Community
Advanced
stablyaistablyai
40.6k

orca-cli

Orchestrate Orca worktrees, terminals, and browser automation via CLI.

Official
Advanced