stat-review

Reviews experiment analyses for power, multiplicity, and assumption violations before shipping decisions.

Updated Sep 11, 2026
One-click install
npx skills add https://github.com/JYeswak/grok_bot_playground --skill stat-review-jyeswak
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: stat-review
Source: https://github.com/JYeswak/grok_bot_playground/tree/main/plugin/skills/stat-review
Command: npx skills add https://github.com/JYeswak/grok_bot_playground --skill stat-review-jyeswak

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Teams routinely ship decisions on analyses that are underpowered, riddled with uncorrected multiple comparisons, or built on unchecked assumptions. This Skill turns each analysis into an explicit verdict — ship, ship with caveats, or do not ship — with the failing check named, so a weak number cannot hide behind a confident headline. ## Core Features & Use Cases - Three mandatory checks: Every review verifies statistical power, multiplicity (all comparisons run, not just reported ones), and method assumptions before any verdict is written. - Verdict with a named reason: Each analysis receives ship, ship with caveats, or do-not-ship, plus the single check that decided it and the most consequential caveat. - Paired-analysis diffing: When two analyses bear on one decision, the review states which one carries the decision and why, or that neither does. - Use Case: Before a rollout go/no-go, hand the Skill two experiment readouts with claims, windows, and sample sizes; it returns a dated markdown file with two verdicts, flagging an underpowered "no effect" as inconclusive rather than a pass. ## Quick Start Ask the assistant to review these two experiment readouts for statistical soundness and give a ship or do-not-ship verdict on each.

Frequently Asked Questions about stat-review

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I review an A/B test result before shipping?

Provide the claim in one sentence, the data window, sample sizes per arm, the method used, and every comparison run including unreported segments. The review checks power, multiplicity, and assumptions, then returns a ship, ship-with-caveats, or do-not-ship verdict with the deciding reason.

How to check if an experiment was underpowered?

The power check names the claimed effect size and the sample per arm, then judges whether the run could plausibly detect that effect. An underpowered "no effect" result is reported as inconclusive, never as a pass.

What is multiple comparisons correction in experiment analysis?

Multiplicity means every comparison actually run — segments, slices, secondary metrics — must be counted, not just the headline. Uncorrected peeking, segment-hunting, or metric-shopping is named explicitly because the headline p-value is meaningless until the search behind it is priced in.

Can this review replace a statistician for consequential decisions?

No. It is a checklist review that catches common errors, not a professional statistical sign-off. Consequential legal, medical, or financial decisions still require a qualified human, and this review is an input to that call, not a substitute.

What happens if an analysis is missing sample sizes or comparisons?

A claim without a sample size is not reviewable and the reviewer asks for it before starting. Unlisted comparisons are assumed to exist until shown otherwise, and any verdict missing one of the three checks is demoted to do-not-ship.