experiment-tracking

To track model training runs effectively, record standardized experiment metadata by generating per-run notebooks and a project-level experiments index. This logs configuration diffs against previous runs for regression visibility and comparability.

Logging experiment metrics like Brier score and ECE requires updating a project-level experiments.md table with consistent measurement values. This ensures every training run maintains standardized calibration and accuracy metrics for post-hoc analysis.

Creating a reproducible experiment notebook involves generating a self-contained exp_NNN_<slug>.ipynb file under your project's experiments directory. This notebook can be rerun from a clean kernel to validate training configurations.

A new experiment tracking notebook should be created only when tracked training knobs differ from the previous experiment. If configurations match, you should instead mark the existing experiment's status as running, done, or failed as appropriate.

Yes, experiment tracking works without external dependencies for local model training. It relies on generating standard notebooks and markdown index files to maintain a single source-of-truth for your training history.

Inconsistent metric deltas in your experiments index occur when fixed delta sign conventions are not applied. Updating experiments.md with consistent metrics and standardized sign conventions ensures accurate regression visibility across runs.