skills.rest
chunkydotdev
chunkydotdevCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

ab-testing

Design statistically valid email A/B tests with sample size calculations and z-test analysis.

2|Updated Mar 30, 2026
Search Tags:#conversion optimization#statistical significance#subject line testing#email ab testing#send time optimization#holdout groups#email experimentation
One-click install
npx skills add https://github.com/chunkydotdev/email-skills --skill ab-testing-chunkydotdev
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ab-testing
Source: https://github.com/chunkydotdev/email-skills/tree/main/skills/content/ab-testing
Command: npx skills add https://github.com/chunkydotdev/email-skills --skill ab-testing-chunkydotdev

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Most email A/B tests produce misleading results due to small sample sizes, early result peeking, and lack of statistical rigor, leading teams to implement changes that do not actually improve performance and waste valuable sending volume.

Core Features & Use Cases

  • Statistical rigor guidance: Includes sample size calculation tables, two-proportion z-test methodology, and confidence interval interpretation to ensure test results are reliable.
  • End-to-end test design: Covers what to test (subject lines, CTAs, send times, content), how long to run tests, randomization best practices, and how to avoid common testing mistakes.
  • Advanced testing methodologies: Explains when to use multivariate testing, bandit algorithms, and holdout groups to measure true incremental lift of email programs. Use case: If your email open rates have plateaued, use this skill to design a valid subject line A/B test with correct sample sizes, avoid peeking at early results, and confidently roll out the winning variant to your full list.

Quick Start

Use the ab-testing skill to design and analyze a statistically valid A/B test for your next promotional email's subject line and CTA combinations.

Frequently Asked Questions about ab-testing

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I calculate sample size for email A/B testing to ensure statistical significance?▼

Calculating sample size for email A/B testing requires using sample size calculation tables and two-proportion z-test methodology to ensure statistical significance. This skill provides the necessary guidance to prevent invalid results from insufficient data volumes.

Why does my email A/B test produce misleading results?▼

Email A/B tests produce misleading results due to small sample sizes, early result peeking, and lack of statistical rigor. Implementing test design guardrails and confidence interval interpretation ensures defensible, actionable experiment outcomes.

What is the best way to test email subject lines and send time optimization?▼

The best way to test email subject lines and send times is through end-to-end test design covering randomization best practices and avoiding common testing mistakes. This ensures reliable measurement of true campaign incremental lift.

Can I use holdout groups to measure true incremental lift in email marketing?▼

You can use holdout groups to measure true incremental lift in email marketing. This skill explains holdout group implementation methodology alongside multivariate testing and bandit algorithms for advanced email experimentation scenarios.

When do I need statistical rigor for automated email sequence optimization?▼

You need statistical rigor for automated email sequence optimization when measuring content variants and CTAs. Applying two-proportion z-test significance analysis and confidence interval interpretation produces defensible experiment outcomes.

Related Skills

Explore More Marketing & Sales Skills→
executiveusaexecutiveusa
1

ab-test-setup

Plan A/B tests with sample sizes, traffic allocation, and implementation checklists.

Community
Intermediate
kmshihab7878kmshihab7878
6

market-emails

Generate complete email sequences with subject lines, timing, and segmentation strategies.

Community
Advanced
1479089714790897
7

kwp-marketing-email-sequence

Design multi-email marketing sequences with copy, timing, and branching logic.

Community
Intermediate
GuitarAlchemistGuitarAlchemist

ix-bandit

Simulate multi-armed bandit algorithms with epsilon-greedy, UCB1, and Thompson sampling.

Official
Intermediate