running-eval-before-after-finetune

Paired statistics tests like McNemar and Wilcoxon compare base versus fine-tuned predictions on the same held-out eval set. This determines whether checkpoint improvements are statistically defensible rather than random variance.

For paired binary classification evaluation, use the McNemar test to compare base versus fine-tuned predictions on the same held-out eval set. This verifies whether observed classification improvements are statistically significant.

To check if fine-tune evaluation results are underpowered, run power checks alongside effect size calculations with 95% confidence intervals. Underpowered results are flagged as inconclusive rather than certifying false improvements.

You need the Wilcoxon signed-rank test for model comparison when evaluating paired-continuous metrics and normality assumptions fail. The evaluation automatically switches from a paired t-test to Wilcoxon based on the assumption check.

Yes, you can compare multiple fine-tuned checkpoints against a base model on the same held-out eval set. The analysis applies appropriate omnibus follow-ups to handle multi-checkpoint scenarios across paired-binary and paired-continuous metrics.

A model comparison refuses to certify checkpoint improvement when row alignment validation fails or a baseline comparison is missing. It also withholds certification for underpowered results, labeling them inconclusive instead of overstating them.