training-check

To monitor WandB training runs for NaN losses, you periodically read training metrics like loss trends, NaN/Inf values, and gradient norms to detect anomalies early and prevent wasted GPU hours on broken runs.

Automated health checking for ML training reliability is the process of periodically monitoring metrics like loss trends, NaN/Inf values, and learning rate behavior to catch issues early before wasting compute resources.

You set up periodic scheduling to check WandB run status by configuring CronCreate to run automated health checks for a specific wandb run at regular intervals, ensuring ongoing monitoring without manual intervention.

Training-check monitors training metrics like loss trends and NaN/Inf values, operating in tandem with watchdog.py to distinguish training quality from process health and covering idle or crashed runs.

For ambiguous training metric signals detected during automated monitoring, Codex MCP handles escalation to provide judgment on unclear cases, ensuring accurate anomaly detection for WandB runs.