skills.rest
czh-ee-2023
czh-ee-2023COMMUNITY
Explore all skills by this author →
WebsiteGitHub

training-check

Monitor WandB training metrics for NaN/Inf, loss divergence, and idle GPUs.

Updated Jun 7, 2026
Search Tags:#machine learning#WandB#training monitoring#Codex integration#training health
One-click install
npx skills add https://github.com/czh-ee-2023/zotero-aris --skill training-check-czh-ee-2023
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: training-check
Source: https://github.com/czh-ee-2023/zotero-aris/tree/main/.claude/skills/training-check
Command: npx skills add https://github.com/czh-ee-2023/zotero-aris --skill training-check-czh-ee-2023

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

This Skill unit helps to identify and address issues during training by monitoring WandB metrics, preventing wasted GPU hours on problematic runs.

Core Features & Use Cases

  • Training Monitoring: Regularly checks WandB metrics to detect anomalies like NaN/Inf values, loss divergence, and idle GPUs.
  • Automated Decision Making: Provides immediate action based on the detected metrics, including stopping training when issues are detected.
  • Codex Integration: Uses Codex for judgment in ambiguous cases to provide a second opinion.

Quick Start

To initiate the training check for a specific wandb run, use the command: Run /training-check for wandb run <entity>/<project>/<run_id>

Frequently Asked Questions about training-check

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I monitor machine learning training health to prevent wasted GPU hours?▼

Detecting NaN/Inf values during model training requires periodic WandB API metric checks. This monitoring process identifies mathematical anomalies and loss divergence in real-time, allowing automated workflows to stop problematic runs immediately and prevent wasted GPU hours.

How do I automatically stop training when WandB metrics show loss divergence?▼

Automated decision-making stops training when WandB metrics show loss divergence by integrating periodic API checks into the workflow. The system evaluates anomalies and triggers immediate action to halt problematic runs without manual intervention.

Does training monitoring with WandB require a specific Python environment setup?▼

Training monitoring with WandB requires a Python environment and wandb API access. You must provide the wandb run identifier in the format <entity>/<project>/<run_id> to initiate the health check for a specific training run.

What is the best way to detect idle GPUs during automated machine learning workflows?▼

Detecting idle GPUs during automated machine learning workflows is best achieved through periodic WandB metric checks. This monitoring approach tracks GPU utilization and identifies idle states, ensuring training health and preventing wasted compute resources.

Can I use Codex integration to evaluate ambiguous training metrics?▼

Codex integration evaluates ambiguous training metrics by providing a second opinion during WandB health checks. The system uses Codex for judgment in unclear cases, ensuring accurate automated decision-making when anomalies are detected.

Related Skills

Explore More Software Engineering Skills→
KwongFukKwongFuk

training-check

Detect WandB metric anomalies during active model training sessions.

Community
Intermediate
junhyekhjunhyekh
1

rlxp-monitor-training

Monitor active RL training runs for failures and metric regressions.

Community
Advanced
zhouboyu-xrealzhouboyu-xreal

weights-and-biases

Log machine learning experiments and hyperparameter sweeps with the wandb library.

Community
Advanced
hve4638hve4638

training-check

Monitors WandB metrics during ML training to detect anomalies and escalate uncertainty.

Community
Intermediate