running-eval-before-after-finetune

What problem does it solve?

This Skill determines whether a fine-tuned checkpoint is genuinely better than its base model by comparing both on the same held-out evaluation set with paired statistical tests, effect sizes, and power checks.

Core Features & Use Cases

It handles paired-binary classification with McNemar tests, paired-continuous metrics with an assumption check that switches between paired t-test and Wilcoxon signed-rank, and multi-checkpoint comparisons with appropriate omnibus follow-ups. It also verifies row alignment, refuses to certify improvement without a baseline comparison, reports effect sizes with 95% confidence intervals, and flags underpowered results as inconclusive rather than overstating them.

Quick Start

Use the running-eval-before-after-finetune skill to compare your base and fine-tuned predictions on the same held-out eval set and tell you whether the improvement is statistically defensible.