training-check

Monitor WandB metrics during model training to detect anomalies.

1|Updated Mar 26, 2026

Search Tags:#automation #monitoring #mlops #metrics #training #anomalies #wandb

Ready to run

npx skills add https://github.com/Lingrongye/federated-learning --skill training-check-lingrongye

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: training-check
Source: https://github.com/Lingrongye/federated-learning/tree/main/Auto-claude-code-research-in-sleep/skills/training-check
Command: npx skills add https://github.com/Lingrongye/federated-learning --skill training-check-lingrongye

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Monitors WandB metrics during training to detect anomalies such as NaN, stalled sessions, or deteriorating loss trends, enabling proactive intervention.

Core Features & Use Cases

Periodic WandB metric checks during active training sessions to identify anomalies early
Fallback to log-file analysis when WandB API is unavailable
Integrates with CronCreate for automated, interval-based monitoring

Quick Start

Configure the training-check to monitor WandB metrics at regular intervals and trigger alerts on anomalies.

Frequently Asked Questions about training-check

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I monitor WandB metrics to detect training anomalies automatically?▼

You can monitor WandB metrics to detect training anomalies by scheduling periodic checks that identify issues like NaN losses or stalled sessions early. This process integrates with CronCreate for automated, interval-based monitoring during active experiments.

What is the best way to catch stalled model training sessions or deteriorating loss trends?▼

The best way to catch stalled model training sessions or deteriorating loss trends is to run periodic WandB metric checks with automated escalation. This approach identifies deteriorating loss trends early, enabling proactive intervention before resources are wasted.

Can I check WandB training metrics if the API is unavailable?▼

Yes, you can check WandB training metrics if the API is unavailable by using a fallback to log-file analysis. This ensures continuous monitoring of training sessions and anomaly detection even when direct API access fails.

How do I set up an automated MLOps monitoring check interval for WandB?▼

You can set up an automated MLOps monitoring check interval for WandB by integrating with watchdog and CronCreate. This allows you to adjust the check interval from 10 to 60 minutes, ensuring periodic metric monitoring during active training.

When should I escalate ambiguous WandB anomaly signals during model training?▼

You should escalate ambiguous WandB anomaly signals during model training when periodic checks return unclear results. The system applies Codex-based judgment to evaluate these ambiguous signals and determine if proactive intervention is necessary.