skills.rest
KYRIE66nb
KYRIE66nbCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

training-check

Monitor WandB metrics during training to detect anomalies and adjust check intervals.

Updated Apr 8, 2026
Search Tags:#automation#metrics#health-check#anomaly-detection#wandb#ml-ops#training-monitoring
One-click install
npx skills add https://github.com/KYRIE66nb/codex-omx-public-config --skill training-check-kyrie66nb
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: training-check
Source: https://github.com/KYRIE66nb/codex-omx-public-config/tree/main/home/.codex/skills/training-check
Command: npx skills add https://github.com/KYRIE66nb/codex-omx-public-config --skill training-check-kyrie66nb

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Periodically read WandB metrics during training to catch problems early and avoid wasting GPU hours on broken runs.

Core Features & Use Cases

  • Automated monitoring of WandB metrics to detect NaN losses, stalled progress, or declining eval metrics.
  • Adaptive scheduling that starts with 10 minutes between checks and increases the interval when health is consistently good.
  • Ambiguity handling via a secondary reviewer to decide between STOP, CONTINUE, or WAIT.
  • Integration with watchdog-style monitoring to surface timely health signals and guide training workflow decisions.

Quick Start

Run /training-check <entity>/<project>/<run_id> to start recurring health checks.

Frequently Asked Questions about training-check

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automatically detect anomalies and stalled progress during WandB model training?▼

To automatically detect anomalies during WandB model training, you can use a health-check tool that periodically reads loss trends and eval metrics to catch NaN losses or stalled progress early.

How does adaptive interval scheduling work for training monitoring?▼

Adaptive interval scheduling for training monitoring starts with 10-minute checks and increases the interval to 20, 30, or 60 minutes when model health is consistently good, reducing unnecessary overhead.

Can I use a secondary reviewer model to handle ambiguous training metrics?▼

Yes, you can use a secondary reviewer model to evaluate ambiguous training metrics, which determines whether to issue a STOP, CONTINUE, or WAIT decision for the active run.

What do I need to set up automated WandB training checks?▼

To set up automated WandB training checks, you need WandB access, a defined interval schedule of 10, 20, 30, or 60 minutes, and a reviewer model to handle ambiguous anomaly detection cases.

Why should I monitor eval metrics to prevent wasted GPU hours?▼

Monitoring eval metrics prevents wasted GPU hours by catching broken runs early through automated anomaly detection, allowing you to stop training immediately when metrics decline or stall.

What is the best way to stop a failing model training run before it wastes resources?▼

The best way to stop a failing model training run is to implement automated health checks that return actionable decisions like STOP, CONTINUE, or WAIT based on real-time WandB metric tracking.

Related Skills

Explore More Data & Analytics Skills→
raphaelchristiraphaelchristi
43

harness:setup

Configure Harness Evolver, LangSmith, and baseline evaluations for Python projects.

Community
Advanced
junhyekhjunhyekh
1

rlxp-monitor-training

Monitor active RL training runs for failures and metric regressions.

Community
Advanced
goupup-aigoupup-ai
2

training-check

Monitor WandB training metrics to detect NaN loss and divergence.

Community
Intermediate
sunLeeesunLeee

convention-monitoring-alerting

Define monitoring and alerting conventions for production systems.

Community
Advanced