skills.rest
satsuki-64
satsuki-64COMMUNITY
Explore all skills by this author →
GitHub

training-check

Monitor WandB metrics during training to detect anomalies and decide whether to continue or halt.

2|Updated Mar 29, 2026
Search Tags:#automation#monitoring#metrics#training#health-check#wandb#ml-debugging
One-click install
npx skills add https://github.com/satsuki-64/MiniAgentWorkflow --skill training-check-satsuki-64
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: training-check
Source: https://github.com/satsuki-64/MiniAgentWorkflow/tree/main/.skills/training-check
Command: npx skills add https://github.com/satsuki-64/MiniAgentWorkflow --skill training-check-satsuki-64

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Periodically monitor WandB metrics during training to detect problems early.

Core Features & Use Cases

  • Reads WandB histories to evaluate loss trends, NaN/Inf occurrences, and gradient behavior.
  • Integrates with CronCreate for scheduled checks and falls back to SSH log inspection if the API is unavailable.
  • Provides clear decisions to continue, wait, or stop training with documented evidence.

Quick Start

Schedule periodic WandB metric reviews during active training to detect anomalies early.

Frequently Asked Questions about training-check

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I monitor WandB metrics to detect training anomalies early?▼

You can monitor WandB metrics by periodically reading history data to evaluate loss trends, NaN/Inf occurrences, and gradient behavior, ensuring early detection of training anomalies.

What is the best way to automate health checks during an active machine learning training session?▼

The best way to automate health checks is by scheduling periodic WandB metric reviews using CronCreate during active training sessions to detect anomalies early.

How do I check training loss trends if the WandB API is unavailable?▼

If the WandB API is unavailable, you can fall back to SSH log inspection to read training data and assess loss trends or anomalies.

Can I automatically stop training when NaN or Inf gradients occur?▼

Yes, by assessing loss trends and gradient behavior, the system evaluates anomalies like NaN/Inf occurrences and triggers actions to continue, wait, or halt training with documented evidence.

How does the system handle ambiguous metrics during a training health check?▼

During a health check, ambiguous metrics are escalated to Codex for further assessment before triggering an action to continue or halt the training process.

Related Skills

Explore More Data & Analytics Skills→
raja21068raja21068
2

training-check

Monitor WandB training metrics to detect NaN, divergence, or degraded evaluation.

Community
Intermediate
duypham2801duypham2801

training-check

Analyze WandB metrics to detect NaNs, loss divergence, and evaluation degradation.

Community
Intermediate
Mang30Mang30

training-check

Detect training quality failures in Weights & Biases runs.

Community
Advanced
hve4638hve4638

training-check

Monitors WandB metrics during ML training to detect anomalies and escalate uncertainty.

Community
Intermediate