skills.rest
meloShaya
meloShayaCOMMUNITY
Explore all skills by this author →
GitHub

experiment-design

Plans single-variable cold email experiments with sample sizes and success criteria.

Updated May 28, 2025
Search Tags:#experiment design#cold email#a/b testing#sample size#campaign optimization#smartlead#reply rate
One-click install
npx skills add https://github.com/meloShaya/insteltech --skill experiment-design-meloshaya
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: experiment-design
Source: https://github.com/meloShaya/insteltech/tree/main/crm/library/skills/experiment-design
Command: npx skills add https://github.com/meloShaya/insteltech --skill experiment-design-meloshaya

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Cold email operators often change the list, copy, and offer at the same time, which destroys all learning signal. This Skill enforces single-variable experiment design so every campaign test produces an attributable, trustworthy result. ## Core Features & Use Cases - Three Experiment Types: Defines list-only, copy-only, and combined experiments with explicit confidence weighting for each. - Sample Size & Success Criteria: Provides minimum sends-per-arm tables based on baseline reply rate and expected lift, plus pre-registered success/failure thresholds. - Structured Output: Produces a YAML experiment plan file with hypothesis, constants, arms, and results fields for tracking through day-21 measurement. - Use Case: You want to test whether Heads of Marketing respond better than VP Sales. The Skill walks you through writing a one-sentence hypothesis, locking all other variables constant, calculating that you need roughly 2,000 sends per arm, and defining the win threshold before launch. ## Quick Start Ask the AI to design a single-variable experiment testing a new target list against your current baseline campaign.

Frequently Asked Questions about experiment-design

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a single-variable cold email experiment?▼

Write a one-sentence hypothesis, change exactly one variable such as the list or copy, and keep everything else constant including offer, infrastructure, and schedule. Launch both arms simultaneously and measure positive reply rate after 21 days.

How many sends do I need per arm for a cold email A/B test?▼

Sample size depends on baseline and expected lift: detecting a 2x lift from a 1% baseline needs about 500 sends per arm, while a 1.5x lift needs about 2,000. Tests under 500 sends per arm cannot separate signal from noise.

Should I use Smartlead's built-in A/B variant feature for testing?▼

Built-in A/B variants mix data within one campaign and are only suitable for small copy tweaks. For real hypothesis testing, create two separate campaigns, split your inboxes evenly, and launch both at the same time.

When should I not run a cold email experiment?▼

Do not experiment until you have a baseline campaign running for at least 3 weeks with a reply rate of 1% or higher after 200 sends. If the baseline is broken, fix deliverability first because experiments on a broken baseline only show that both arms are bad.

Why wait 21 days before measuring cold email experiment results?▼

Cold email replies trickle in over the full sequence, typically Day 0, 3, 7, and 11 plus a reply grace period. Measuring earlier biases results toward the first email's reply rate and misses later-sequence conversions.

Related Skills

Explore More Marketing & Sales Skills→
PinkycherryPinkycherry

prospeo-search-api

Search leads with the Prospeo Search Person API using filters, rate limiting, and state-by-state crawling.

Community
Intermediate
PinkycherryPinkycherry

experiment-design

Plan single-variable cold email experiments with sample sizes and success criteria.

Community
Intermediate
PinkycherryPinkycherry

email-deliverability-audit

Diagnose cold email deliverability by auditing SPF, DKIM, DMARC, inbox health, and bounce rates via Smartlead.

Community
Advanced
athina-aiathina-ai
1.1k

meta-ads-campaign-builder

Generates Meta Ads campaign structures for Facebook and Instagram.

Official
Advanced