training-check

Name: training-check
Availability: InStock
Author: tqli

Monitor WandB metrics to detect NaN, divergence, and idle GPUs during training runs.

Updated Aug 23, 2026

Search Tags:#metrics #gpu #anomaly-detection #wandb #ml-ops #training-monitoring

Ready to run

npx skills add https://github.com/tqLi99/academic-paper-skills --skill training-check

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: training-check
Source: https://github.com/tqLi99/academic-paper-skills/tree/main/skills/training-check
Command: npx skills add https://github.com/tqLi99/academic-paper-skills --skill training-check

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Periodically read WandB metrics during training to catch problems early, preventing wasted GPU hours from unnoticed issues such as NaN, divergence, or idle resources.

Core Features & Use Cases

WandB metric monitoring and alerting during long-running experiments.
Configurable check intervals (starting at 10 minutes, increasing when healthy).
Automatic proceed for healthy runs and a secondary reviewer for ambiguous cases.

Quick Start

Start monitoring a live training run by providing the entity/project/run_id or by deriving it from project notes, and let the skill handle health checks automatically.

Frequently Asked Questions about training-check

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I monitor WandB metrics to catch training divergence or NaN issues early?▼

Monitor WandB metrics in real time to detect problematic training runs by checking for NaN, loss divergence, and metric degradation at configurable intervals, preventing wasted GPU hours during long-running experiments.

What is the best way to detect idle GPUs during long-running ML experiments?▼

Detect idle GPUs during long-running ML experiments by periodically reading WandB metrics to monitor training health, which identifies inactive resources and prevents wasted compute on unnoticed failures.

Can I automatically proceed with healthy training runs while monitoring WandB metrics?▼

Yes, automatically proceed with healthy training runs using the AUTO_PROCEED feature, which increases check intervals when metrics are stable and only triggers alerts for ambiguous cases using a secondary reviewer model.

Do I need to pass an explicit run_id to monitor training with WandB, or can it be derived automatically?▼

You do not need to pass an explicit run_id to monitor training with WandB; the skill can derive the entity, project, and run_id directly from project notes as needed, or accept them explicitly.

How does anomaly detection work for ML training monitoring with WandB?▼

Anomaly detection for ML training monitoring works by periodically reading WandB metrics to identify loss divergence, NaN values, and metric degradation, using a secondary reviewer model to assess ambiguous results.

What should I do if my training loss diverges during a long-running experiment?▼

If training loss diverges during a long-running experiment, use real-time WandB metric monitoring to catch the divergence early, which prevents wasted GPU hours by alerting you before the run completes.