skills.rest
kitcaf
kitcafCOMMUNITY
Explore all skills by this author →
GitHub

training-check

Monitor WandB metrics periodically to detect training issues like NaN/Inf or stagnation.

1|Updated Mar 24, 2026
Search Tags:#monitoring#metrics#gpu#training#health-check#wandb#machine-learning
One-click install
npx skills add https://github.com/kitcaf/skills --skill training-check-kitcaf
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: training-check
Source: https://github.com/kitcaf/skills/tree/main/skills/skills-codex/skills/training-check
Command: npx skills add https://github.com/kitcaf/skills --skill training-check-kitcaf

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Periodically read WandB metrics during training to catch problems early. Do not wait until training finishes to discover it was a waste of GPU time.

Core Features & Use Cases

  • Read WandB metrics programmatically to assess training health and detect issues like NaN/Inf, stagnation, or sudden changes in learning rate.
  • Provide structured judgement signals and escalation paths, enabling automated responses or human review when needed.
  • Use with recurring scheduling to continuously monitor live runs and protect GPU resources.

Quick Start

Create a recurring job that runs /training-check <entity>/<project>/<run_id> every 10 minutes.

Frequently Asked Questions about training-check

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I monitor live model training metrics to detect issues early?▼

Automated health checks monitor WandB metrics periodically during active training to detect issues like NaN, Inf, or stagnation early, preventing wasted GPU time by alerting you before the run finishes.

What is the best way to automate health checks for multiple WandB runs?▼

The best way to automate health checks for multiple runs is scheduling a recurring job that queries your entity, project, and run_id at a configurable interval, providing structured judgement signals and escalation paths for automated responses or human review.

Do I need a WandB API key and run_id to check training health?▼

Yes, you need WandB API access and a defined run_id or entity/project/run_id combination to programmatically read metrics and assess training health for your live machine learning runs.

Can I detect learning rate anomalies and NaN metrics during active machine learning training?▼

Yes, you can detect learning rate anomalies and NaN metrics during active training by reading WandB metrics programmatically to identify sudden changes, numerical instabilities, or stagnation in your live runs.

How do I stop wasting GPU time on failing model training runs?▼

You can stop wasting GPU resources by setting up recurring monitoring jobs to validate WandB metrics every few minutes, allowing you to catch and escalate issues like stagnation or NaN values early instead of waiting for training to finish.

Related Skills

Explore More Software Engineering Skills→
bglick13bglick13
1

experiment-analysis

Diagnose GRPO training runs using WandB metrics and Axiom logs.

Community
Advanced
ys6448761-hueys6448761-hue

comi-orchestrator

Centralize Aurora5 progress tracking and generate automated daily status reports.

Community
Advanced
AstorYHAstorYH
15

system-health-monitor

Monitor CPU, memory, and disk space against predefined thresholds.

Community
Intermediate
HeXiao-55HeXiao-55
1

training-check

Monitor WandB metrics to detect NaN, divergence, or stalled GPU utilization.

Community
Intermediate