hyperparameter-tuning

Define an objective function that uses trial.suggest_int, suggest_float, and suggest_categorical to sample hyperparameters, then train and score the model. Create a study with optuna.create_study(direction='maximize') and call study.optimize with your trial budget.

Use grid search for small spaces under 100 trials, random search for large spaces with limited budget since it explores more unique configurations, and Bayesian optimization for expensive evaluations like deep learning where guided sampling saves compute.

Optuna pruning uses integration callbacks like XGBoostPruningCallback that report intermediate validation metrics during training. Trials performing worse than historical medians are stopped early, freeing compute for more promising configurations.

Yes, learning rates span orders of magnitude from 1e-5 to 1e-1, so use log=True in suggest_float or tune.loguniform. Linear sampling clusters values near the upper bound and misses small learning rates that often matter most.

Nested cross-validation tunes hyperparameters on an inner loop and evaluates on an untouched outer test fold, preventing optimistic bias. Standard tuning on a single CV split overestimates generalization performance because the test data influenced parameter selection.

Around 100 trials is often sufficient for most models, and 1000 or more is rarely needed. Set early stopping patience so tuning halts when scores stop improving, and prioritize high-impact parameters like learning rate, depth, and regularization first.