skills.rest
delta-and-beta
delta-and-betaCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

tester-experimentation

Design and evaluate A/B experiments to validate campaign and feature changes.

Updated Apr 6, 2026
Search Tags:#ab-testing#statistical-analysis#experiment-design#optimizely#vwo#eppo#kameleoon
One-click install
npx skills add https://github.com/delta-and-beta/braze-agency --skill tester-experimentation
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tester-experimentation
Source: https://github.com/delta-and-beta/braze-agency/tree/main/skills/tester-experimentation
Command: npx skills add https://github.com/delta-and-beta/braze-agency --skill tester-experimentation

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps teams design, instrument, and interpret A/B tests and experiments across major platforms so they can reliably determine which campaign or feature variant improves target business metrics while avoiding common pitfalls like SRM and peeking.

Core Features & Use Cases

  • Cross-platform guidance: Platform-specific workflows for VWO, Optimizely, Kameleoon, and Eppo, including integration notes for Braze and data warehouses.
  • Experiment design & statistics: MDE/sample size calculation, sequential vs Bayesian testing, CUPED variance reduction, and guardrail metrics.
  • Variant allocation & rollout: Best practices for sticky bucketing, exclusion groups, ramp strategies, and production graduation of winners.
  • Use case: Validate a new personalized email variation in a Canvas campaign, monitor SRM and guardrails, analyze results in a warehouse or partner stats engine, and decide whether to graduate the winner.

Quick Start

Design an A/B test comparing control and treatment, calculate the required sample size and ramp plan, instrument exposure and outcome events in Braze or your warehouse, and monitor SRM and guardrail metrics until a statistically sound decision can be made.

Frequently Asked Questions about tester-experimentation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I calculate sample size and MDE for an A/B test?▼

To calculate sample size and MDE for an A/B test, you define the minimum detectable effect and baseline conversion rate, then apply statistical models to determine the required traffic volume per variant before launching the experiment.

What is SRM detection and why does it matter in experiment analysis?▼

SRM detection identifies sample ratio mismatch in A/B testing, alerting you when actual traffic allocation differs significantly from expected ratios, which prevents drawing false conclusions from corrupted experiment data.

How do I set up variant allocation and sticky bucketing in Optimizely or VWO?▼

Variant allocation and sticky bucketing in Optimizely or VWO involve assigning users to experiment variants consistently using persistent identifiers, ensuring they see the same variation across sessions while maintaining proper exclusion groups and ramp strategies.

Can I use CUPED variance reduction with warehouse-native experimentation in Eppo?▼

Yes, you can use CUPED variance reduction with Eppo's warehouse-native experimentation to leverage pre-experiment covariates, reducing variance and increasing statistical sensitivity without requiring additional client-side platform integrations.

Does this A/B testing workflow integrate with Braze and data warehouses?▼

Yes, this A/B testing workflow integrates with Braze and data warehouses, allowing you to instrument exposure and outcome events, monitor guardrail metrics, and analyze experiment results directly within your existing warehouse environment.

When should I use Bayesian testing versus sequential testing for campaign optimization?▼

Use Bayesian testing for campaign optimization when you need continuous probability updates and intuitive results, and sequential testing when you want to monitor experiments continuously with statistical validity while avoiding peeking pitfalls.

Related Skills

Explore More Marketing & Sales Skills→
delta-and-betadelta-and-beta

architect-data-pipelines

Design data pipeline architectures for Braze Decisioning Studio with governance and reverse ETL.

Community
Advanced
masharrattmasharratt
14

CFN Marketing Landing Pages

Automate landing-page creation, A/B testing, and publishing via shell scripts and webhooks.

Community
Advanced
delta-and-betadelta-and-beta

strategist-feature-flags

Orchestrate Braze feature flag rollouts and Canvas experiments with YAML frontmatter.

Community
Advanced
weisbergweisberg
1

experimentation

Automate A/B experiment design, sequential monitoring, CUPED analysis, and HTML reporting.

Community
Advanced