training-check

Monitors WandB metrics during ML training to detect anomalies and escalate uncertainty.

Updated Mar 1, 2026

Search Tags:#monitoring #training #ml #anomaly-detection #quality-checks #wandb

Ready to run

npx skills add https://github.com/hve4638/hve-cc-marketplace --skill training-check-hve4638

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: training-check
Source: https://github.com/hve4638/hve-cc-marketplace/tree/main/aris/skills/skills-codex/training-check
Command: npx skills add https://github.com/hve4638/hve-cc-marketplace --skill training-check-hve4638

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Periodically read WandB metrics during training to catch problems early, helping you avoid wasting GPU hours by discovering issues before long-running runs finish.

Core Features & Use Cases

Periodic WandB metric reads to detect anomalies such as NaN/Inf values, stagnation, or divergence.
Escalation workflow via a secondary Codex reviewer when metrics are ambiguous, with adaptive check intervals.
Use cases include long-running experiments, hyperparameter sweeps, and monitoring training health in production-like environments.

Quick Start

Run the training-health check against a live training run to begin periodic assessments.

Frequently Asked Questions about training-check

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I monitor WandB metrics to detect NaN values and loss divergence during ML training?▼

To monitor WandB metrics for anomaly detection during ML training, this Skill automates periodic evaluation of active sessions to detect NaN values, loss divergence, or plateauing metrics early. It reads WandB history including loss trends, eval metrics, and gradient norms to catch issues before long-running runs finish.

What ML training health checks can I automate to catch issues early and save GPU hours?▼

You can automate ML training health checks that periodically read WandB metrics to catch problems like NaN or Inf values, stagnation, and divergence early. This ongoing quality check helps you avoid wasting GPU hours by discovering issues before long-running experiments finish.

How does the escalation reviewer work when WandB metrics are ambiguous?▼

When WandB metrics are ambiguous during training health checks, the Skill triggers an escalation workflow via a secondary Codex reviewer. It also adapts the check interval dynamically, ensuring uncertain anomaly detection results receive further scrutiny without constant manual monitoring.

Can I use this for monitoring training health in hyperparameter sweeps and long-running experiments?▼

Yes, monitoring training health in hyperparameter sweeps and long-running experiments is a core use case. The Skill applies periodic quality checks to active training sessions, reading signals like eval metrics, gradient norms, and learning rate to detect early anomalies.

Do I need any specific dependencies to run periodic WandB evaluations for anomaly detection?▼

No specific dependencies are required to run periodic WandB evaluations for anomaly detection. The Skill operates standalone to read WandB history and signals such as loss trends, eval metrics, NaN or Inf values, gradient norms, and learning rate during your ML training.