Monitor WandB training metrics to detect NaN loss and divergence.
npx skills add https://github.com/goupup-ai/miccai25 --skill training-check-goupup-ai
Please help me install this Agent Skill. Skill: training-check Source: https://github.com/goupup-ai/miccai25/tree/main/ARIS/skills/training-check Command: npx skills add https://github.com/goupup-ai/miccai25 --skill training-check-goupup-ai
Long-running machine learning training runs can fail silently due to issues like NaN loss, loss divergence, or metric degradation, often wasting hours of expensive GPU compute before the problem is discovered. This Skill automates periodic checks of training metrics to catch these issues early, preventing wasted resources and reducing debugging time.
Use the training-check skill to run an automated health check on your active WandB training run <entity>/<project>/<run_id> and flag any critical issues like NaN loss or metric degradation.
High-intent search queries and answers about installing and using this skill.
To monitor machine learning training runs for silent failures, you can automate periodic checks of WandB metrics to detect NaN loss, loss divergence, and metric degradation before they waste GPU compute resources.
Detect NaN loss and loss divergence early by applying automated periodic health checks to your WandB metrics, which flag critical issues immediately and escalate ambiguous metric changes to a reasoning model.
Yes, you can use WandB to track training health alongside process watchdog tools to cover both runtime stability, such as checking if the session is alive and GPU is active, and overall training quality.
The best way to save GPU hours when model training fails silently is to implement automated metric monitoring that stops runs early upon detecting metric degradation or loss divergence, minimizing manual intervention.
Automated training monitoring works for multi-day sessions on cloud GPU infrastructure by providing low-overhead, configurable health checks that catch silent failures and prevent wasted compute time on broken experiments.
Mang30Detect training quality failures in Weights & Biases runs.
GonglitianDiagnose ML training issues from symptom to root cause across GPU setups.
xqinagMonitor WandB metrics during training to detect NaN, divergence, or idle GPUs.
Wenwen555Monitor WandB metrics during training to detect NaN values, loss divergence, and idle GPUs.
