skills.rest
kalilurrahman
kalilurrahmanCOMMUNITY
Explore all skills by this author →
Website𝕏@krahmanGitHub

ab-test-ml

Design and analyze A/B tests for ML models with statistical significance testing and traffic routing.

1|Updated Mar 21, 2026
Search Tags:#experiment design#mlflow#power analysis#ab testing#statistical significance#traffic splitting#ml models
One-click install
npx skills add https://github.com/kalilurrahman/kr-claudiator-skills-original-prompts --skill ab-test-ml-kalilurrahman
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ab-test-ml
Source: https://github.com/kalilurrahman/kr-claudiator-skills-original-prompts/tree/main/04-ai-ml/ab-test-ml
Command: npx skills add https://github.com/kalilurrahman/kr-claudiator-skills-original-prompts --skill ab-test-ml-kalilurrahman

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires numpy, scipy, pandas, statsmodels, mlflow.

What problem does it solve? Deploying a new ML model without rigorous experimentation risks shipping regressions that hurt business metrics. This Skill provides a complete framework for designing, running, and analyzing A/B tests on production ML models so deployment decisions are grounded in statistical evidence rather than intuition. ## Core Features & Use Cases - Experiment Design & Power Analysis: Calculate required sample sizes and estimated runtime using statsmodels before launching any test. - Traffic Splitting & Model Routing: Route users to champion or challenger models with deterministic hash-based sticky assignment, including shadow mode for zero-risk comparison. - Statistical Analysis & Guardrails: Run two-sample t-tests, compute confidence intervals, and block shipping when guardrail metrics like revenue or latency degrade. - Use Case: Your team trains a new recommendation model. Use this Skill to size the experiment, route 10% of users to the challenger via MLflow-loaded models, and generate a ship/rollback recommendation after 14 days. ## Quick Start Design an A/B test for my new recommendation model with a 5% CTR improvement target and 100,000 daily users, then analyze the results.

Frequently Asked Questions about ab-test-ml

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I A/B test a machine learning model in production?▼

Define a hypothesis and one primary business metric, run power analysis to size the experiment, then route traffic with deterministic hash-based assignment so users see a consistent variant. Analyze results with a two-sample t-test only after the planned runtime completes.

How to calculate sample size for an A/B test in Python?▼

Use statsmodels power analysis classes like NormalIndPower for proportion metrics or TTestIndPower for continuous metrics. Provide the baseline value, minimum detectable effect, significance level of 0.05, and power of 0.80 to get the required sample size per arm.

What is shadow mode deployment for ML models?▼

Shadow mode runs both the control and treatment models on live traffic but only returns the control output to users. It lets you compare prediction distributions and detect issues without any production risk before starting a real A/B test.

Why should I not stop an A/B test early when results look significant?▼

P-values fluctuate over time, so peeking and stopping early inflates false positive rates. Run the full planned duration from your power analysis, or use sequential testing methods designed for continuous monitoring.

What are guardrail metrics in ML experiments?▼

Guardrail metrics are secondary measurements like revenue per user, error rate, and p99 latency that must not degrade during an experiment. A statistically significant drop in any guardrail blocks shipping even if the primary metric improves.

How does sticky assignment work in A/B testing?▼

Sticky assignment hashes the experiment ID and user ID to deterministically place each user in the same variant on every request. This prevents users from flip-flopping between control and treatment, which would contaminate the results.

Related Skills

Explore More Data & Analytics Skills→
organvm-i-theoriaorganvm-i-theoria

ml-experiment-tracker

Track ML experiments, version datasets, and manage model registries with MLflow, W&B, and DVC.

Official
Intermediate
maminul007maminul007

model-registry

Manage AI model versions across staging and production environments.

Community
Advanced
tonone-aitonone-ai
69

cortex-eval

Evaluate model performance, drift, and error patterns on production data.

Official
Advanced
GuitarAlchemistGuitarAlchemist

ix-bandit

Simulate multi-armed bandit algorithms with epsilon-greedy, UCB1, and Thompson sampling.

Official
Intermediate