comparing-models-fairly

What problem does it solve?

This Skill prevents misleading model comparisons by replacing eyeballed metric differences with the correct paired statistical test and decision rule.

Core Features & Use Cases

• Paired significance testing: Chooses McNemar, DeLong, paired t-test, Wilcoxon signed-rank, or Friedman/Nemenyi based on how the models were evaluated.
• Fair comparison discipline: Enforces same-test-set validation, paired cross-validation folds, and correction for three or more models.
• Decision-ready reporting: Requires p-values, effect sizes, confidence intervals, and an operational threshold before declaring a winner.
• Use case: Compare two classifiers on the same held-out set, rank several cross-validated candidates, or reject an invalid unpaired comparison workflow.

Quick Start

Ask the skill to compare your models on the same data and return the correct paired test, significance result, effect size, and correction-aware recommendation.