training-check

Monitor WandB metrics during training to detect NaN, divergence, or idle GPUs.

Updated Jun 10, 2026

Search Tags:#codex #training #cron #health-check #wandb #ml-monitoring #gpu-efficiency

Ready to run

npx skills add https://github.com/xqinag/ARIS-new --skill training-check-xqinag

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: training-check
Source: https://github.com/xqinag/ARIS-new/tree/main/skills/training-check
Command: npx skills add https://github.com/xqinag/ARIS-new --skill training-check-xqinag

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Periodically monitor WandB metrics during training to detect problems early and avoid wasting GPU hours.

Core Features & Use Cases

Automated health checks during training to detect NaN, divergence, or idle GPUs.
Fallback to log file analysis if WandB API is unreachable.
Adjustable cadence with Codex-assisted judgment for ambiguous signals.

Quick Start

Schedule periodic training-health checks to monitor metrics and trigger alerts on anomalies.

Frequently Asked Questions about training-check

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I monitor WandB metrics to detect NaN or divergence during training?▼

To monitor WandB metrics, you can schedule periodic health checks that automatically analyze training runs to detect anomalies like NaN, divergence, or idle GPUs. This catches issues early to prevent wasting GPU hours.

What is the best way to set up automated health checks for ongoing ML training runs?▼

Automated health checks for ML training runs use a configurable cadence to periodically evaluate WandB metrics. This process detects problems early, preventing wasted GPU hours by identifying issues like divergence or idle hardware.

Can I analyze training logs for anomalies if the WandB API is unreachable?▼

Yes, you can analyze training logs for anomalies if the WandB API is unreachable. The monitoring process supports an optional fallback to log file analysis to ensure continuous health checks during training.

How does Codex assist with ambiguous signals in ML monitoring?▼

Codex assists with ambiguous signals in ML monitoring by providing automated escalation and judgment. It evaluates uncertain metric patterns detected during periodic health checks to determine if a training issue exists.

Do I need a specific cron setup to run periodic health checks on my GPUs?▼

You need a cron setup to schedule periodic health checks on your GPUs. This configurable cadence automates the monitoring of WandB metrics, ensuring continuous detection of training issues like idle hardware or divergence.