skills.rest
dz306271098
dz306271098COMMUNITY
Explore all skills by this author →
GitHub

training-check

Monitors WandB metrics to detect training issues and stop or adjust runs.

Updated Apr 2, 2026
Search Tags:#automation#monitoring#training#ml#health-check#wandb
One-click install
npx skills add https://github.com/dz306271098/ARIS_for_Robotics --skill training-check-dz306271098
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: training-check
Source: https://github.com/dz306271098/ARIS_for_Robotics/tree/main/skills/training-check
Command: npx skills add https://github.com/dz306271098/ARIS_for_Robotics --skill training-check-dz306271098

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Periodically read WandB metrics during training to catch problems early. Do not wait until training finishes to discover it was a waste of GPU time.

Core Features & Use Cases

  • Read WandB metrics programmatically from WandB API to monitor training runs.
  • Evaluate signals such as loss trends, eval metrics, NaN/Inf values, spikes, and learning rate schedules.
  • Escalate ambiguous cases to Codex judgment and act on clear signals (Stop, Continue, Wait).
  • Integrate with a watchdog workflow and CronCreate automation to ensure ongoing health checks.

Quick Start

Configure a recurring health check that reads the WandB run metrics and triggers alerts when anomalies are detected.

Frequently Asked Questions about training-check

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I monitor WandB metrics to detect ML training issues early?▼

Monitor WandB metrics by periodically reading run data to detect training issues early, evaluating loss trends, eval metrics, NaN/Inf values, and gradient spikes against predefined thresholds. This prevents wasted GPU hours by stopping or adjusting runs based on data-driven signals.

Can I automate WandB health checks to stop training runs automatically?▼

Yes, you can automate WandB health checks using CronCreate and a watchdog workflow. The system reads run metrics and applies predefined thresholds to automatically stop, continue, or wait based on clear data-driven signals.

What is the best way to handle ambiguous ML training metrics detected via WandB?▼

The best way to handle ambiguous ML training metrics is to escalate the cases to Codex judgment. The system acts directly on clear signals like Stop or Continue, but routes uncertain data trends for further evaluation.

What specific training anomalies can I track by reading WandB metrics programmatically?▼

By reading WandB metrics programmatically, you can track loss trends, eval metrics, NaN/Inf values, gradient spikes, and learning rate schedules. This allows you to catch problems early rather than waiting until training finishes.

Do I need CronCreate to set up recurring WandB run health checks?▼

Yes, integrating with CronCreate is required to configure a recurring health check. This integration automates ongoing monitoring, reading WandB run metrics and triggering alerts when anomalies are detected.

Related Skills

Explore More Data & Analytics Skills→
KwongFukKwongFuk

training-check

Detect WandB metric anomalies during active model training sessions.

Community
Intermediate
goupup-aigoupup-ai
2

training-check

Monitor WandB training metrics to detect NaN loss and divergence.

Community
Intermediate
LingrongyeLingrongye
1

training-check

Monitor WandB metrics during model training to detect anomalies.

Community
Advanced
jandan138jandan138

training-check

Monitor WandB training metrics for anomalies and crashed runs.

Community
Advanced