training-check

Monitor WandB metrics to detect NaN, divergence, or stalled GPU utilization.

1|Updated Mar 25, 2026

Search Tags:#automation #cron #wandb #ml-training #gpu-utilization #training-monitoring

Ready to run

npx skills add https://github.com/HeXiao-55/Auto-SurveyMind --skill training-check-hexiao-55

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: training-check
Source: https://github.com/HeXiao-55/Auto-SurveyMind/tree/main/skills/training-check
Command: npx skills add https://github.com/HeXiao-55/Auto-SurveyMind --skill training-check-hexiao-55

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Periodic WandB monitoring catches training issues early, preventing wasted GPU hours and failed experiments by surfacing anomalous trends before they derail the run.

Core Features & Use Cases

Real-time WandB metric monitoring for anomalies such as NaN, divergence, and stalled runs.
Configurable cadence with auto-escalation guidance and optional Codex-based judgment steps.
Automation hooks to notify, pause, or stop training when issues are detected, enabling rapid remediation.

Quick Start

Start the monitoring loop during an active training run to automatically report suspicious trends and escalate when issues are detected.

Frequently Asked Questions about training-check

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I monitor WandB metrics in real time to catch training issues?▼

To monitor WandB metrics in real time, start the automated checking loop during an active training run to detect anomalies like NaN or divergence and receive actionable escalation guidance.

How can I automate alerts for stalled GPU utilization or NaN values during model training?▼

Automate alerts for stalled GPU utilization or NaN values by configuring a scheduled cadence with auto-escalation guidance, using automation hooks to notify, pause, or stop training when issues are detected.

What is the best way to prevent wasted GPU hours from failed machine learning experiments?▼

The best way to prevent wasted GPU hours is applying periodic WandB monitoring to surface anomalous trends early, catching training issues before they derail the run and trigger automated interventions.

Can I trigger automated interventions or pause training when WandB metrics show divergence?▼

Yes, you can trigger automated interventions or pause training when WandB metrics show divergence by configuring automation hooks that execute auto-escalation workflows during active model training.

Do I need a scheduled cadence and WandB access to start automated training checks?▼

Yes, you need WandB access and a scheduled cadence to start automated training checks, along with an escalation workflow that can trigger automated interventions or Codex judgments.

Why does my model training run diverge before the WandB monitoring system alerts me?▼

Model training diverges before alerts if the scheduled cadence is too sparse; applying real-time WandB metric monitoring catches suspicious trends earlier, preventing failed experiments from stalling GPU utilization.