training-check

Detect loss divergence or NaN values by periodically checking WandB training metrics to spot training quality failures early. This automated monitoring reads loss and evaluation trends during an active run to identify problems before completion, saving GPU hours.

Yes, you can automatically stop training when metrics degrade. The monitoring applies automated triage with clear actions: it stops training for clearly bad signals, continues for healthy trends, and escalates to an LLM judgment only when ambiguous.

If WandB is unreachable, training monitoring falls back to reading the training log file to avoid false assumptions. This fail-safe mechanism ensures automated health checks continue identifying loss divergence or degraded evaluation metrics without active WandB access.

Yes, you need a WandB run identified as entity/project/run_id to monitor training quality. This identifier provides the access required to periodically check loss and evaluation metrics for NaN, Inf, or divergence issues during an active deep learning run.

Yes, you can optionally compare active training metrics against a baseline reference run. This comparison helps verify experiment quality by evaluating whether current evaluation metrics degrade relative to the established baseline during automated periodic health checks.