training-check

Monitor WandB metrics in real time to detect problematic training runs by checking for NaN, loss divergence, and metric degradation at configurable intervals, preventing wasted GPU hours during long-running experiments.

Detect idle GPUs during long-running ML experiments by periodically reading WandB metrics to monitor training health, which identifies inactive resources and prevents wasted compute on unnoticed failures.

Yes, automatically proceed with healthy training runs using the AUTO_PROCEED feature, which increases check intervals when metrics are stable and only triggers alerts for ambiguous cases using a secondary reviewer model.

You do not need to pass an explicit run_id to monitor training with WandB; the skill can derive the entity, project, and run_id directly from project notes as needed, or accept them explicitly.

Anomaly detection for ML training monitoring works by periodically reading WandB metrics to identify loss divergence, NaN values, and metric degradation, using a secondary reviewer model to assess ambiguous results.

If training loss diverges during a long-running experiment, use real-time WandB metric monitoring to catch the divergence early, which prevents wasted GPU hours by alerting you before the run completes.