experiment-analyze

What problem does it solve?

This Skill automates post-experiment analysis by reading experiment outputs, running a statistical audit, comparing observed outcomes against pre-registered hypotheses and benchmarks, and synthesizing a finding that surfaces reproducibility and failure-mode issues for human review.

Core Features & Use Cases

• Statistical Audit: Runs scripts/audit_stats.py to compute trials per condition, confidence intervals, effect sizes, variance, and flags statistical insufficiency or missing baselines.
• Mode-Aware Analysis: Supports reproduction, diagnostic, and approach modes to compute per-condition verdicts, reproducibility = {pass|partial|fail}, hypothesis status, and mechanism checks (ablations vs priors).
• Governed Action & Provenance: Appends experiment_analysis and finding records, proposes backward triggers (t4/t5/t7/t8/t14/t15) but requires human confirmation before any state transition.
• Use Case: After an experiment finishes, run this Skill to validate statistical rigor, detect new failure modes, compare to benchmarks, and produce a concise, reviewable finding with recommended next steps.

Quick Start

Run the experiment-analyze skill on the experiment directory to audit statistics, compare outcomes to hypotheses and benchmarks, and produce a finding with proposed backward triggers for human review.