skills.rest
lix965996-art
lix965996-artCOMMUNITY
Explore all skills by this author →
GitHub

training-check

Continuously evaluate WandB metrics to detect training quality failures.

1|Updated May 14, 2026
Search Tags:#mlops#anomaly detection#wandb#gpu training#training monitoring#cron automation#loss divergence
One-click install
npx skills add https://github.com/lix965996-art/MMM --skill training-check-lix965996-art
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: training-check
Source: https://github.com/lix965996-art/MMM/tree/main/resources/app/skills/training-check
Command: npx skills add https://github.com/lix965996-art/MMM --skill training-check-lix965996-art

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps you detect training quality issues early so you do not waste GPU hours on runs that are going wrong (for example NaNs, loss divergence, or degrading evaluation metrics).

Core Features & Use Cases

  • Periodically reviews WandB training signals: monitors loss trends, evaluation metric changes, learning-rate behavior, gradient norm stability, and NaN/Inf occurrences.
  • Makes a risk-based decision: classifies signals as clearly bad, clearly fine, or unsure, escalating only when ambiguous.
  • Safe fallback when WandB is unavailable: if WandB cannot be reached, it can fall back to reading a training log via SSH to avoid false assumptions.

Use case: You kick off a long multi-hour training job and want automated health checks that catch subtle quality failures before the run finishes.

Quick Start

Run the training-check skill for the WandB run path wandb-run-path and let it monitor metrics on its configured interval.

Frequently Asked Questions about training-check

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I detect loss divergence or NaNs during a long GPU training run?▼

Training monitoring evaluates WandB metrics like loss trends, NaNs, and gradient norms during active runs to classify risk and stop training or escalate only when signals are ambiguous.

How do I monitor WandB metrics automatically on a schedule for my ML training jobs?▼

You can monitor WandB metrics automatically by providing a run path in the form entity/project/run_id, applying interval backoff with anomaly reset to periodically check training health.

Does training monitoring still work if the WandB API is unreachable?▼

Yes, if WandB is unreachable, the monitoring falls back to reading a training log via SSH to avoid false assumptions about the training run's health.

Can I automatically stop GPU training when evaluation metrics degrade?▼

Yes, the monitoring makes risk-based decisions by classifying signals as clearly bad, clearly fine, or unsure, and it stops training automatically when signals are clearly bad to save GPU hours.

What ML training signals should I check to prevent wasted GPU hours?▼

To prevent wasted GPU hours, check loss trends, evaluation metric changes, learning-rate behavior, gradient norm stability, and NaN/Inf occurrences periodically during the training run.

Related Skills

Explore More Software Engineering Skills→
duypham2801duypham2801

training-check

Analyze WandB metrics to detect NaNs, loss divergence, and evaluation degradation.

Community
Intermediate
xqinagxqinag

training-check

Monitor WandB metrics during training to detect NaN, divergence, or idle GPUs.

Community
Advanced
ovachieverovachiever
52

weights-and-biases

Automate ML experiment tracking and model registry management via the wandb API.

Community
Advanced
esopianesopian
2

canary

Monitor live applications by comparing screenshots and logs against baselines.

Community
Intermediate