optuna-nested-cv

What problem does it solve?

It eliminates unreliable hyperparameter tuning by running a persistent Optuna search inside a nested cross-validation structure, producing an unbiased generalization estimate while preventing holdout leakage.

Core Features & Use Cases

• Nested CV for unbiased evaluation: Uses an outer holdout fold for evaluation and an inner Optuna study to tune only on the outer training split.
• Resumable SQLite-backed studies: Ensures trials are auditable and restartable via a stable study name and persistent storage.
• Scalar objective + composite scoring: Supports single-scalar optimization (including composite objectives) to keep selection unambiguous.
• Split discipline (tune vs holdout): Enforces explicit bank roles so the search never “peeks” at the evaluation data.
• Sampler-policy contract: Keeps sampling regimes consistent between search and final training, so results transfer correctly.
• Use in tuning expensive pipelines: Designed for costly evaluations where the search budget must be efficiently managed and results must generalize.

Quick Start

Use the optuna-nested-cv skill to run an Optuna TPE hyperparameter search over an inner loop for each outer CV fold, persisting trials to sqlite and reporting the mean outer-fold score as the unbiased estimate.