skills.rest
HeXiao-55
HeXiao-55COMMUNITY
Explore all skills by this author →
WebsiteGitHub

training-check

Monitor WandB metrics to detect NaN, divergence, or stalled GPU utilization.

1|Updated Mar 25, 2026
Search Tags:#automation#cron#wandb#ml-training#gpu-utilization#training-monitoring
One-click install
npx skills add https://github.com/HeXiao-55/Auto-SurveyMind --skill training-check-hexiao-55
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: training-check
Source: https://github.com/HeXiao-55/Auto-SurveyMind/tree/main/skills/training-check
Command: npx skills add https://github.com/HeXiao-55/Auto-SurveyMind --skill training-check-hexiao-55

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Periodic WandB monitoring catches training issues early, preventing wasted GPU hours and failed experiments by surfacing anomalous trends before they derail the run.

Core Features & Use Cases

  • Real-time WandB metric monitoring for anomalies such as NaN, divergence, and stalled runs.
  • Configurable cadence with auto-escalation guidance and optional Codex-based judgment steps.
  • Automation hooks to notify, pause, or stop training when issues are detected, enabling rapid remediation.

Quick Start

Start the monitoring loop during an active training run to automatically report suspicious trends and escalate when issues are detected.

Frequently Asked Questions about training-check

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I monitor WandB metrics in real time to catch training issues?▼

To monitor WandB metrics in real time, start the automated checking loop during an active training run to detect anomalies like NaN or divergence and receive actionable escalation guidance.

How can I automate alerts for stalled GPU utilization or NaN values during model training?▼

Automate alerts for stalled GPU utilization or NaN values by configuring a scheduled cadence with auto-escalation guidance, using automation hooks to notify, pause, or stop training when issues are detected.

What is the best way to prevent wasted GPU hours from failed machine learning experiments?▼

The best way to prevent wasted GPU hours is applying periodic WandB monitoring to surface anomalous trends early, catching training issues before they derail the run and trigger automated interventions.

Can I trigger automated interventions or pause training when WandB metrics show divergence?▼

Yes, you can trigger automated interventions or pause training when WandB metrics show divergence by configuring automation hooks that execute auto-escalation workflows during active model training.

Do I need a scheduled cadence and WandB access to start automated training checks?▼

Yes, you need WandB access and a scheduled cadence to start automated training checks, along with an escalation workflow that can trigger automated interventions or Codex judgments.

Why does my model training run diverge before the WandB monitoring system alerts me?▼

Model training diverges before alerts if the scheduled cadence is too sparse; applying real-time WandB metric monitoring catches suspicious trends earlier, preventing failed experiments from stalling GPU utilization.

Related Skills

Explore More Data & Analytics Skills→
KwongFukKwongFuk

training-check

Detect WandB metric anomalies during active model training sessions.

Community
Intermediate
dz306271098dz306271098

training-check

Monitors WandB metrics to detect training issues and stop or adjust runs.

Community
Advanced
KappTech88KappTech88

weights-and-biases

Track ML experiments with automatic logging and manage model registry via Weights & Biases.

Community
Advanced
goupup-aigoupup-ai
2

training-check

Monitor WandB training metrics to detect NaN loss and divergence.

Community
Intermediate