training-check

Monitor WandB metrics during training to detect anomalies and decide whether to continue or halt.

2|Updated Mar 29, 2026

Search Tags:#automation #monitoring #metrics #training #health-check #wandb #ml-debugging

Ready to run

npx skills add https://github.com/satsuki-64/MiniAgentWorkflow --skill training-check-satsuki-64

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: training-check
Source: https://github.com/satsuki-64/MiniAgentWorkflow/tree/main/.skills/training-check
Command: npx skills add https://github.com/satsuki-64/MiniAgentWorkflow --skill training-check-satsuki-64

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Periodically monitor WandB metrics during training to detect problems early.

Core Features & Use Cases

Reads WandB histories to evaluate loss trends, NaN/Inf occurrences, and gradient behavior.
Integrates with CronCreate for scheduled checks and falls back to SSH log inspection if the API is unavailable.
Provides clear decisions to continue, wait, or stop training with documented evidence.

Quick Start

Schedule periodic WandB metric reviews during active training to detect anomalies early.

Frequently Asked Questions about training-check

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I monitor WandB metrics to detect training anomalies early?▼

You can monitor WandB metrics by periodically reading history data to evaluate loss trends, NaN/Inf occurrences, and gradient behavior, ensuring early detection of training anomalies.

What is the best way to automate health checks during an active machine learning training session?▼

The best way to automate health checks is by scheduling periodic WandB metric reviews using CronCreate during active training sessions to detect anomalies early.

How do I check training loss trends if the WandB API is unavailable?▼

If the WandB API is unavailable, you can fall back to SSH log inspection to read training data and assess loss trends or anomalies.

Can I automatically stop training when NaN or Inf gradients occur?▼

Yes, by assessing loss trends and gradient behavior, the system evaluates anomalies like NaN/Inf occurrences and triggers actions to continue, wait, or halt training with documented evidence.

How does the system handle ambiguous metrics during a training health check?▼

During a health check, ambiguous metrics are escalated to Codex for further assessment before triggering an action to continue or halt the training process.