tester-experimentation

What problem does it solve?

This Skill helps teams design, instrument, and interpret A/B tests and experiments across major platforms so they can reliably determine which campaign or feature variant improves target business metrics while avoiding common pitfalls like SRM and peeking.

Core Features & Use Cases

• Cross-platform guidance: Platform-specific workflows for VWO, Optimizely, Kameleoon, and Eppo, including integration notes for Braze and data warehouses.
• Experiment design & statistics: MDE/sample size calculation, sequential vs Bayesian testing, CUPED variance reduction, and guardrail metrics.
• Variant allocation & rollout: Best practices for sticky bucketing, exclusion groups, ramp strategies, and production graduation of winners.
• Use case: Validate a new personalized email variation in a Canvas campaign, monitor SRM and guardrails, analyze results in a warehouse or partner stats engine, and decide whether to graduate the winner.

Quick Start

Design an A/B test comparing control and treatment, calculate the required sample size and ramp plan, instrument exposure and outcome events in Braze or your warehouse, and monitor SRM and guardrail metrics until a statistically sound decision can be made.