training-check

You can monitor training metrics by periodically checking WandB logs or fallback files for anomalies like NaN and divergence. This process analyzes training loss trends and eval metrics to detect broken runs and evaluate the overall training health.

Training health checking is the process of analyzing eval metrics, loss trends, and learning rate schedules to ensure effective model training. You need it when running Codex sessions to detect plateaus, prevent wasted compute on broken runs, and verify training continuation validity.

You decide whether to continue or stop training by analyzing WandB logs for loss trends, eval metrics, and learning rate schedules. By checking for plateaus, divergence, or NaN values, you can make clear decisions on training continuation to optimize performance.

Yes, you can check for training plateaus and broken runs using fallback logs when WandB is unavailable. The monitoring process evaluates training loss trends and eval metrics from these local logs to detect anomalies and provide decisions on training continuation.

Yes, training health monitoring works with standard Codex session logs by analyzing them directly or using them as fallback logs. It checks these logs for NaN, divergence, and plateaus to evaluate training quality and determine if the training process should continue.