skills.rest
TabithaFanny
TabithaFannyCOMMUNITY
Explore all skills by this author →
GitHub

training-check

Monitor WandB training metrics for NaN values, loss divergence, and idle GPUs.

Updated May 29, 2026
Search Tags:#monitoring#metrics#health checks#training#WandB
One-click install
npx skills add https://github.com/TabithaFanny/ThesisX --skill training-check-tabithafanny
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: training-check
Source: https://github.com/TabithaFanny/ThesisX/tree/main/skills_imported/aris/skills/training-check
Command: npx skills add https://github.com/TabithaFanny/ThesisX --skill training-check-tabithafanny

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires wandb, and includes scripts (resource) components.

What problem does it solve?

This Skill provides automated health checks during training to catch potential issues like NaN values, loss divergence, or idle GPUs early, saving time and resources.

Core Features & Use Cases

  • Automated WandB Metrics Monitoring: Periodically checks WandB metrics during training for anomalies like NaN values, loss divergence, or sudden spikes.
  • Codex MCP for Ambiguous Cases: Uses Codex MCP to get expert judgment on ambiguous training metrics.
  • Integration with Watchdog: Works alongside watchdog.py to provide comprehensive training session management.
  • Use Case: During deep learning model training, use this Skill to ensure the training process is running smoothly and to catch any issues before they cause significant resource waste.

Quick Start

Set up a Cron job to run the training-check skill for your wandb run 'entity/project/run_id'.

Frequently Asked Questions about training-check

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I monitor WandB metrics to catch NaN values or loss divergence during model training?▼

You can monitor WandB metrics and catch NaN values by running this Skill as a scheduled health check against your training run. It automatically detects anomalies like loss divergence, sudden spikes, or idle GPUs to prevent wasted compute resources.

What is automated training health checking and how does it work with WandB?▼

Automated training health checking works by periodically polling WandB metrics during a live training session to detect anomalies. This Skill utilizes the Codex MCP to evaluate ambiguous metrics and integrates with watchdog.py for comprehensive session management.

Do I need a Codex API key to run automated health checks on my training metrics?▼

Yes, you need Codex API access to run automated health checks on your training metrics. The Skill utilizes Codex MCP specifically to provide expert judgment when encountering ambiguous training metrics that standard threshold checks cannot resolve.

What is the best way to detect idle GPUs and training issues early in a deep learning run?▼

The best way to detect idle GPUs and training issues early is to set up a Cron job that runs this Skill for your specific WandB run. It periodically evaluates your metrics alongside watchdog.py to ensure the training process is running smoothly.

How do I set up a Cron job to check my WandB training run for anomalies?▼

To set up a Cron job to check your WandB run for anomalies, schedule the training-check Skill to execute periodically against your target run identifier. Configure the Cron job to pass your 'entity/project/run_id' as the input for continuous session monitoring.

Can I use watchdog.py with this Skill for comprehensive training session management?▼

Yes, you can use watchdog.py with this Skill for comprehensive training session management. The Skill is designed to work alongside watchdog.py, combining automated WandB metric polling with broader session control to catch issues before wasting resources.

Related Skills

Explore More Data & Analytics Skills→
czh-ee-2023czh-ee-2023

training-check

Monitor WandB training metrics for NaN/Inf, loss divergence, and idle GPUs.

Community
Advanced
winsorllcwinsorllc

health-monitor

Track system metrics and detect anomalies with threshold-based alerting.

Community
Intermediate
jandan138jandan138

training-check

Monitor WandB training metrics for anomalies and crashed runs.

Community
Advanced
Wenwen555Wenwen555

training-check

Monitor WandB metrics during training to detect NaN values, loss divergence, and idle GPUs.

Community
Advanced