skills.rest
weisberg
weisbergCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

analyze-results

Run A/B analysis with SRM checks and statistical tests in Python.

1|Updated Mar 19, 2026
Search Tags:#python#statistics#reproducibility#p-values#ab-testing#confidence-intervals#a-b-test
One-click install
npx skills add https://github.com/weisberg/agile_agentic_analytics --skill analyze-results-weisberg
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: analyze-results
Source: https://github.com/weisberg/agile_agentic_analytics/tree/main/plugins/ab-testing/skills/analyze-results
Command: npx skills add https://github.com/weisberg/agile_agentic_analytics --skill analyze-results-weisberg

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Analyzing A/B test results can be error-prone without structured statistical checks. This skill provides a clear framework to extract reliable conclusions from variable data, guard against common pitfalls, and communicate results effectively.

Core Features & Use Cases

  • SRM check for validating sample allocations and surfacing potential data integrity issues
  • Appropriate statistical tests for proportions, means, and multiple variants with effect sizes and confidence intervals
  • Multi-variant analysis with corrections for multiple testing and plain-English interpretation

Quick Start

Provide your A/B data in CSV format or as a simple table; the skill will run the analysis and return an interpretation.

Frequently Asked Questions about analyze-results

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I analyze A/B test results with multiple variants and correct for multiple testing?▼

To analyze A/B test results with multiple variants, the skill applies corrections for multiple testing, selects appropriate statistical tests, and provides effect sizes with confidence intervals and plain-English interpretation.

What is an SRM check and why is it needed for A/B testing data?▼

An SRM check validates sample allocations in A/B testing to surface potential data integrity issues before analysis. It ensures the observed traffic split matches the expected ratio, preventing flawed conclusions.

How do I calculate p-values and confidence intervals for proportions and means in Python?▼

To calculate p-values and confidence intervals for proportions and means, the skill generates reproducible Python code using scipy.stats and statsmodels, selecting appropriate tests based on your metric type.

Can I use this skill if my A/B testing data is in a simple table instead of CSV format?▼

Yes, you can provide your A/B testing data in CSV format or as a simple table. The skill will identify the metric type, variants, and data format to perform statistically valid analysis.

What is the best way to ensure reproducibility when running statistical tests for A/B analysis?▼

The best way to ensure reproducibility is to use generated Python code with scipy.stats and statsmodels that reproduces the full A/B analysis end-to-end, including SRM checks and effect size calculations.

Related Skills

Explore More Data & Analytics Skills→
jaydubya818jaydubya818

ab-test-setup

Plans and designs A/B tests with hypothesis, metrics, and sample size.

Community
Intermediate
ckallumckallum

layman

Summarize code changes in plain English for product and design stakeholders.

Community
Intermediate
weisbergweisberg
1

sample-size

Compute sample size and experiment duration for A/B tests.

Community
Advanced
guideopsguideops

trading-journal

Automate trade logging, P&L calculation, and performance reporting for traders.

Community
Intermediate