results-analysis

Analyze ML experiment results with rigorous statistics and generate scientific figures.

Updated Oct 7, 2022
One-click install
npx skills add https://github.com/tamagusko/linux-cfg --skill results-analysis-tamagusko
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: results-analysis
Source: https://github.com/tamagusko/linux-cfg/tree/main/dotfiles/claude/skills/results-analysis
Command: npx skills add https://github.com/tamagusko/linux-cfg --skill results-analysis-tamagusko

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Experimental results in ML/AI research are often summarized with cherry-picked best scores, missing significance tests, or decorative figures. This Skill enforces an evidence-first workflow that validates experiment artifacts, runs proper descriptive and inferential statistics, and produces real publication-quality figures with explicit interpretation. ## Core Features & Use Cases - Strict statistical analysis: Computes mean ± std, 95% confidence intervals, significance tests with assumption checks, effect sizes, and multiple-comparison corrections, with non-parametric fallbacks when assumptions fail. - Real scientific figures: Generates actual comparison, ablation, and training-dynamics figures from readable data, each with purpose, caption requirements, and interpretation checklists. - Structured analysis bundle: Produces analysis-report.md, stats-appendix.md, figure-catalog.md, and a figures/ directory, explicitly separating evidence from manuscript prose. - Use Case: You have seed-level CSV results comparing a full fine-tuned model, an adapter variant, and a frozen-encoder baseline. This Skill validates the comparison units, runs paired tests with Holm correction, and outputs a complete analysis bundle ready to hand off to a results-report writer. ## Quick Start Ask the assistant to run a strict statistical analysis of the experiment results in your results directory and generate the full analysis bundle with figures.

Frequently Asked Questions about results-analysis

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I statistically compare machine learning model results across seeds?

Aggregate seed-level metrics, verify the unit of analysis, then run paired tests such as a paired t-test or Wilcoxon signed-rank test after checking normality. Report mean ± std, 95% CI, effect size, and apply corrections like Holm when testing multiple contrasts.

What statistical test should I use for ablation study comparisons?

For paired ablation results, use a paired t-test if normality holds, otherwise Wilcoxon signed-rank. For three or more conditions, use repeated-measures ANOVA or the Friedman test, followed by corrected post-hoc tests such as Tukey HSD or Nemenyi.

When should I use non-parametric tests instead of t-tests?

Use non-parametric tests when Shapiro-Wilk indicates non-normality, sample sizes are very small, or data contain strong outliers. Mann-Whitney U replaces the independent t-test and Wilcoxon replaces the paired t-test in these cases.

Why is reporting only p-values insufficient in experiment analysis?

P-values alone hide effect magnitude and practical significance. Rigorous reporting requires effect sizes such as Cohen's d, confidence intervals, sample sizes, and multiple-comparison corrections so readers can judge whether differences matter.

What happens if my experiment data is incomplete or lacks seed-level runs?

The analysis degrades explicitly: without seed-level data only descriptive summaries are produced and inferential claims are blocked. Missing baselines prevent significance claims, and all blockers are stated rather than replaced with confident prose.

Does this skill write the paper Results section?

No. It produces analysis artifacts such as reports, statistics appendices, and figures, but deliberately excludes manuscript prose. Drafting the Results section is handed off to separate paper-writing or results-report skills.