skills.rest
tswr
tswrCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

ab-testing

Design, launch, and analyze online controlled experiments with statistical rigor.

1|Updated Mar 22, 2026
Search Tags:#conversion rate#hypothesis testing#ab-testing#statistical significance#sample size calculation#online experiments#experimentation pitfalls
One-click install
npx skills add https://github.com/tswr/engineering-mastery-plugin --skill ab-testing-tswr
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ab-testing
Source: https://github.com/tswr/engineering-mastery-plugin/tree/main/skills/ab-testing
Command: npx skills add https://github.com/tswr/engineering-mastery-plugin --skill ab-testing-tswr

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill eliminates the risk of running flawed A/B tests that produce false positives, p-hacked results, and incorrect product decisions, ensuring your experiments yield statistically valid, actionable insights.

Core Features & Use Cases

  • End-to-end experiment design: Guides hypothesis formation, pre-registration, metric selection (overall evaluation criterion, guardrail metrics), and sample size/power calculation to avoid underpowered or biased tests.
  • Robust statistical analysis: Supports both frequentist and Bayesian methods, includes multiple comparison correction, confidence interval reporting, and variance reduction techniques like CUPED.
  • Pitfall mitigation: Detects and avoids common failures including peeking, survivorship bias, Simpson's paradox, Sample Ratio Mismatch, and interference effects.
  • Use case: A product team testing a new checkout flow can use this Skill to design a valid experiment, calculate the required sample size, analyze results correctly, and avoid shipping a change that harms conversion.

Quick Start

Use the ab-testing skill to design a valid A/B test for the new checkout flow, calculate the required sample size, and analyze the results for statistical and practical significance.

Frequently Asked Questions about ab-testing

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I calculate the required sample size for an A/B test to avoid underpowered results?▼

Avoid peeking during an A/B test by pre-registering your hypothesis and committing to a fixed sample size, as continuous monitoring inflates false positives and generates p-hacked results.

How do I detect and mitigate Simpson's paradox in online experiments?▼

Mitigate Simpson's paradox in online experiments by ensuring proper randomization and analyzing segmented user data, preventing flawed product decisions from reversed statistical trends.

Does this A/B testing method support both frequentist and Bayesian statistical analysis?▼

This A/B testing method supports both frequentist and Bayesian analysis, applying multiple comparison correction, confidence interval reporting, and CUPED variance reduction for robust significance testing.

What are the most common A/B testing pitfalls that generate false positive results?▼

Common A/B testing pitfalls generating false positives include peeking at data early, Sample Ratio Mismatch, survivorship bias, and interference effects across user groups.

How do I design a valid online controlled experiment for a new checkout flow?▼

Design a valid online controlled experiment for a checkout flow by pre-registering your hypothesis, selecting guardrail metrics, calculating sample size, and analyzing statistical and practical significance.

Related Skills

Explore More Software Engineering Skills→
sicktasticsicktastic

ab-test-setup

Plan statistically sound A/B tests with sample size and duration calculations.

Community
Advanced
runchr-comrunchr-com

ab-test-analysis

Analyze A/B test results for statistical significance and decision recommendations.

Official
Intermediate
donygeorgedonygeorge
2

plan

Generate milestone-driven implementation plans with testable exit criteria.

Community
Advanced
James-TrainaJames-Traina
11

causal-inference

Implement causal identification strategies like IV/2SLS, DiD, RDD, synthetic control, and matching.

Community
Advanced