experiment-readout

Evaluates reported A/B test results into supported, unsupported, or underpowered verdicts.

Updated Sep 11, 2026
One-click install
npx skills add https://github.com/JYeswak/grok_bot_playground --skill experiment-readout-jyeswak
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: experiment-readout
Source: https://github.com/JYeswak/grok_bot_playground/tree/main/plugin/skills/experiment-readout
Command: npx skills add https://github.com/JYeswak/grok_bot_playground --skill experiment-readout-jyeswak

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Teams routinely make decisions on reported test results that lack effect sizes, confidence intervals, or adequate sample sizes. This Skill turns a reported result into a verdict that is actually supported by the data, before anyone agrees that something "worked". ## Core Features & Use Cases - Structured readout sequence: Reports the effect in decision-relevant units, a 95% interval, the assumptions that must hold, and a final verdict of supported, unsupported, or underpowered. - Underpowered detection: Computes the sample size required for the claimed effect at 80% power, so "not significant" becomes an actionable number instead of a dead end. - Guardrails against misuse: Refuses to report a p-value without an effect size, explains the cost of stopping tests early, and names the specific confounder blocking any causal claim from observational data. - Use Case: A PM reports "conversion rose 2% after the redesign, p=0.08". The Skill returns the effect with its interval, states the test is underpowered, and gives the per-arm sample size needed to detect a 2pp change. ## Quick Start Ask the assistant to evaluate a reported experiment result, for example: "Our A/B test showed a 3% conversion lift with n=200 per arm, is this significant?"

Frequently Asked Questions about experiment-readout

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I know if my A/B test result is significant?▼

Report the effect size with a 95% confidence interval and check whether the interval excludes no-effect. A p-value alone cannot size a decision, so it should always be paired with the effect size in the unit the reader cares about.

What sample size do I need for an A/B test?▼

Required sample size depends on the effect you want to detect and the desired power. For example, n=40 per arm detects a 15 percentage point change at 80% power, while detecting a 3pp change needs roughly 1,700 per arm.

Can I stop an A/B test early when results look good?▼

Stopping early inflates the error rate because repeated peeking increases false positives. The readout explains what early stopping does to the error rate and how many observations the planned power actually requires.

Why is a p-value alone not enough to make a decision?▼

A p-value indicates whether an effect is distinguishable from noise but says nothing about its magnitude. A tiny, meaningless effect can be significant with enough data, so the p-value must be reported with the effect size.

Can observational data prove a change caused the result?▼

No. Observational data shows association, not causation. The readout reports the association and names the specific confounder that would have to be ruled out before any causal claim could hold.