skills.rest
OpenLAIR
OpenLAIROFFICIAL
Explore all skills by this author →
GitHub

aris-training-check

Monitors WandB training metrics periodically to detect NaN, divergence, and idle GPUs.

1.1k|116|Updated Feb 26, 2026
Search Tags:#GPU#WandB#CRON#training monitoring#loss divergence#Machine learning#Health check
One-click install
npx skills add https://github.com/OpenLAIR/dr-claw --skill aris-training-check
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: aris-training-check
Source: https://github.com/OpenLAIR/dr-claw/tree/main/skills/aris-training-check
Command: npx skills add https://github.com/OpenLAIR/dr-claw --skill aris-training-check

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires wandb.

What problem does it solve?

Long training runs can silently waste GPU hours when loss diverges, NaN values appear, or metrics plateau, and these issues are often only discovered after training finishes. This Skill performs periodic automated health checks on running training jobs so problems are caught early.

Core Features & Use Cases

  • Periodic WandB Metric Checks: Reads training loss, eval metrics, gradient norms, and learning rate schedules from the WandB API, with SSH log-file fallback when WandB is unreachable.
  • Tiered Judgment Workflow: Classifies signals as clearly bad (stop training), clearly fine (continue), or ambiguous, escalating only ambiguous cases to a Codex model for a STOP/CONTINUE/WAIT verdict.
  • Adaptive Check Intervals: Starts at 10-minute intervals and increases up to 60 minutes when consistently healthy, resetting after anomalies, and integrates with a watchdog process that handles process-level health.
  • Use Case: After launching a multi-day LLM fine-tuning run, schedule this Skill via CronCreate to automatically inspect WandB metrics every 10 minutes and kill the run with logged evidence if the loss diverges.

Quick Start

Set up a recurring check every 10 minutes to monitor my WandB run at entity/project/run_id and stop training if the loss diverges.

Frequently Asked Questions about aris-training-check

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I monitor WandB training metrics automatically?▼

Use the WandB API to fetch run history and check loss trends, eval metrics, NaN values, and gradient norms on a recurring schedule. This Skill automates that via CronCreate, starting at 10-minute intervals and increasing up to 60 minutes when healthy.

How to detect NaN or loss divergence during model training?▼

Check the last N steps of training loss for NaN or Inf values, sudden spikes over 10x normal variance, and sustained increases across multiple checkpoints. Clearly bad signals trigger an immediate stop; ambiguous trends are escalated to a model for judgment.

What happens if WandB is unreachable during a training check?▼

The Skill falls back to reading the training log directly over SSH using tail on the remote log file. If both WandB and logs are unreachable, it reports the connectivity issue and retries at the next interval rather than assuming training is broken.

What is the difference between training quality checks and process health checks?▼

Process health checks (watchdog.py) verify the session is alive and GPUs are active every 60 seconds. Training quality checks examine loss trends and metric improvement every 10-60 minutes, catching subtle issues like plateaus that process monitoring misses.

When should I not stop a training run based on metrics?▼

Do not stop on the first sign of noise, since some loss spikes are normal. Judge based on trends over multiple checkpoints, and only stop for clearly bad signals like NaN, sustained divergence, or eval metrics significantly worse than baseline.

Related Skills

Explore More Education & Research Skills→
caw111caw111

training-check

Monitor Codex training metrics for NaN, divergence, plateaus, and broken runs.

Community
Advanced
DifanaDAPDifanaDAP

weights-and-biases

Automate ML experiment tracking and hyperparameter sweeps via the Python API.

Community
Advanced
goupup-aigoupup-ai
2

training-check

Monitor WandB training metrics to detect NaN loss and divergence.

Community
Intermediate
LingrongyeLingrongye
1

training-check

Monitor WandB metrics during model training to detect anomalies.

Community
Advanced