skills.rest
lightrain-a
lightrain-aCOMMUNITY
Explore all skills by this author →
GitHub

training-check

Monitor WandB training metrics to detect loss divergence, NaN/Inf, and eval degradation.

Updated May 20, 2026
Search Tags:#early stopping#wandb#model quality#training monitoring#metric anomaly detection#cron automation#gpu cost control
One-click install
npx skills add https://github.com/lightrain-a/medtrace-aris --skill training-check-lightrain-a
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: training-check
Source: https://github.com/lightrain-a/medtrace-aris/tree/main/.vendor/aris/skills/training-check
Command: npx skills add https://github.com/lightrain-a/medtrace-aris --skill training-check-lightrain-a

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

It prevents wasted GPU time by detecting early training quality failures such as loss divergence, NaNs/Infs, and degrading evaluation metrics.

Core Features & Use Cases

  • Periodic WandB metric health checks: Continuously evaluates loss trends and evaluation metric movement while training is still running.
  • Clear decision rules with escalation: Stops for clearly bad signals, continues when clearly healthy, and escalates ambiguous cases to a Codex judgment step.
  • Fallback when WandB is unreachable: If WandB cannot be accessed, it checks the training log via SSH to avoid false assumptions about failure.

Use case example: During multi-hour model training on expensive hardware, run this check every 10–60 minutes to stop promptly if loss diverges or metrics collapse, instead of discovering the issue only after training completes.

Quick Start

Set up a recurring CronCreate job to run training-check against the WandB run path so it can automatically monitor the run and decide whether to stop, continue, or wait.

Frequently Asked Questions about training-check

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I stop bad model training runs early to save GPU hours?▼

Detect model training anomalies early by monitoring WandB metrics for loss divergence, NaNs, Infs, and degrading evaluation metrics. This early stopping mechanism checks run health at timed intervals and halts wasted GPU hours before completion.

How does metric anomaly detection work during active training?▼

Metric anomaly detection works by applying interval backoff to check WandB training metrics every 10 to 60 minutes. It evaluates loss trends, eval metric movement, and learning-rate behavior, escalating ambiguous signals to a Codex judgment step for STOP, CONTINUE, or WAIT decisions.

Can I monitor training logs via SSH if WandB is unreachable?▼

You can monitor training logs via SSH if WandB is unreachable. The system falls back to tailing logs over an SSH connection to check training health, avoiding false failure assumptions when the metric tracking platform is inaccessible.

Do I need a WandB run path to automate training monitoring?▼

You need a WandB run path containing the entity, project, and run ID to automate training monitoring via a recurring CronCreate job. This setup enables automated periodic health checks against your active metrics.

What is the best way to control GPU costs during multi-hour model training?▼

Control GPU costs during multi-hour model training by running automated periodic checks for metric anomalies. This approach uses interval backoff and clear decision rules to stop runs promptly if loss diverges, avoiding the cost of discovering failures after completion.

Related Skills

Explore More Software Engineering Skills→
duypham2801duypham2801

training-check

Analyze WandB metrics to detect NaNs, loss divergence, and evaluation degradation.

Community
Intermediate
openscientist-ioopenscientist-io
44

stopping-criteria

Evaluate iteration state and budget constraints to decide whether to continue or synthesize.

Official
Advanced
jandan138jandan138

training-check

Monitor WandB training metrics for anomalies and crashed runs.

Community
Advanced
HeXiao-55HeXiao-55
1

training-check

Monitor WandB metrics to detect NaN, divergence, or stalled GPU utilization.

Community
Intermediate