experiment-design

Plan single-variable cold email experiments with sample sizes and success criteria.

Updated Aug 2, 2026
One-click install
npx skills add https://github.com/Pinkycherry/newbusinessideas3 --skill experiment-design-pinkycherry
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: experiment-design
Source: https://github.com/Pinkycherry/newbusinessideas3/tree/main/.claude/skills/experiment-design
Command: npx skills add https://github.com/Pinkycherry/newbusinessideas3 --skill experiment-design-pinkycherry

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Changing your list, copy, and offer at the same time destroys all learning signal in cold email campaigns. This Skill forces you to isolate one variable per experiment so you actually know what drove the result. ## Core Features & Use Cases - Three Experiment Types: Defines list-only, copy-only, and combined experiments with confidence weighting for each type of learning. - Sample Size & Baseline Rules: Provides minimum sends-per-arm tables, a 1% baseline sanity check, and a 21-day measurement window before evaluating results. - Structured Experiment Plan: Outputs a YAML experiment file with hypothesis, constants, success criteria, and arms for control and variant campaigns. - Use Case: You want to test whether Heads of Marketing respond better than VP Sales. The Skill walks you through locking copy and offer constant, calculating 2,000 sends per arm, launching both campaigns simultaneously in Smartlead, and measuring positive reply rate at day 21. ## Quick Start Ask the AI to design a list-only experiment testing a new target job title against your current baseline campaign.

Frequently Asked Questions about experiment-design

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I A/B test cold email copy correctly?

Run a copy-only experiment: keep the list, offer, and sending infrastructure identical, and change only the copy. Launch two separate campaigns in Smartlead or Instantly at the same time, then compare positive reply rates after 21 days.

How many sends do I need for a cold email experiment?

Plan for at least 500 sends per arm to detect a 2x lift from a 1% baseline, and around 2,000 sends per arm for a 1.5x lift. Fewer than 500 sends per arm makes it impossible to separate signal from noise.

Why shouldn't I change my list and copy at the same time?

Changing multiple variables at once destroys attribution: if results improve or regress, you cannot tell whether the list, copy, or offer caused it. Isolate one variable per experiment, or explicitly classify it as a combined experiment with only medium-confidence learnings.

When should I measure cold email experiment results?

Measure at day 21, after the full sequence (typically Day 0, 3, 7, 11) plus a reply grace period completes for all leads. Measuring earlier biases results toward the first email's reply rate.

What baseline do I need before running cold email experiments?

Confirm an overall reply rate of at least 1% after 200 or more sends before experimenting. Below that threshold, your infrastructure or copy is already broken, so run a deliverability audit first instead of testing variants.