training-check

To monitor WandB training runs for loss divergence and NaN detection, you can use a scheduled health check that periodically queries WandB metrics to evaluate loss trends, learning-rate scheduling behavior, and numerical stability in near-real time.

Loss divergence detection works by periodically querying WandB history to analyze loss trends and eval metric changes during ongoing training, applying rule-based triage to automatically stop runs when bad signals are identified.

The best way to automatically stop a machine learning training job when metrics degrade is to run scheduled health checks that evaluate WandB metrics, capturing the run URL and key metrics as evidence before stopping the run.

Yes, you can check training quality if WandB is unreachable. The monitoring process includes a resilient fallback that reads the training log remotely via SSH to continue evaluating training health.

To handle ambiguous eval metric changes during continuous training workflows, the system escalates uncertain signals to an MCP Codex judgment, while directly stopping or continuing the run for clearly bad or good signals.

A limitation to detecting numerical stability issues during long training jobs is that monitoring requires a periodic WandB history query, and ambiguous signals may need escalation to MCP Codex judgment, which can delay the stop decision.