skills.rest
PostHog
PostHogOFFICIAL
Explore all skills by this author →
Website𝕏@PostHogGitHub

debugging-experiments

Diagnose PostHog experiment result anomalies and produce customer-facing explanations with fixes.

713|118|Updated Aug 11, 2020
Search Tags:#support#Debugging#feature-flags#experiments#AB-testing#PostHog#sample-ratio-mismatch
One-click install
npx skills add https://github.com/PostHog/posthog-foss --skill debugging-experiments
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: debugging-experiments
Source: https://github.com/PostHog/posthog-foss/tree/main/products/experiments/skills/debugging-experiments
Command: npx skills add https://github.com/PostHog/posthog-foss --skill debugging-experiments

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

Support engineers handling PostHog Experiments tickets need to explain why a customer's A/B test results look wrong — uneven variant splits, missing exposures, bias banners, or numbers that don't match SQL — and this Skill turns those complaints into a data-backed diagnosis and a ready-to-send customer reply.

Core Features & Use Cases

  • Known-cause catalog: Matches complaints like uneven exposures, zero-traffic variants, and missing exposures to ranked root causes such as multiple-variant exclusion bias, sample ratio mismatch, capture-by-surface, and wrong SDK accessor methods.
  • Read-only data pull sequence: Pulls experiment config, per-variant exposure totals, exposure shape via HogQL, and flag change history through PostHog MCP tools before asking the customer anything.
  • SRM localization script: Ships srm_check.py, which recomputes PostHog's deterministic flag hash offline and decomposes a confirmed sample ratio mismatch into assignment-side versus capture-side causes.
  • Customer reply templates: Converts internal findings into plain-language replies using the exact UI labels customers see, with links to the experiment, flag, and cohort on the correct US or EU instance.
  • Use Case: A customer pastes a ticket asking why one variant gets no traffic; the Skill pulls the experiment's real data, identifies a forced-variant release condition, and drafts the explanation and fix.

Quick Start

Debug this experiment support ticket where the customer says their variant exposures are uneven and the bias banner is showing.

Frequently Asked Questions about debugging-experiments

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I debug uneven variant exposures in a PostHog experiment?▼

Pull the experiment config and per-variant exposure totals, then run a chi-squared test on per-person counts against the configured split. Common causes are the multiple-variant exclusion bias, capture-by-surface skew, or users reading the flag before it loads.

Why is my PostHog experiment showing zero or too few exposures?▼

Missing exposures usually come from using bulk SDK accessors like getFeatureFlagPayload instead of single-flag accessors, disabled send_feature_flag_events, a global holdout siphoning users, or a custom exposure event missing the variant property.

What causes the sample ratio mismatch bias banner in PostHog experiments?▼

The banner appears when an uneven variant split combines with Exclude from analysis handling and users exposed to more than one variant, dropping the smaller variant's users asymmetrically. Switching to Use first seen variant or an even split resolves it.

How do I tell if an experiment skew is assignment-side or capture-side?▼

Recompute each user's variant from PostHog's deterministic flag hash offline and decompose the gap between recorded and configured splits into selection and reassignment components. The bundled srm_check.py script runs this test and names the side only when one component dominates.

When should I not use this experiment debugging workflow?▼

Do not use it for creating experiments, configuring rollout or metrics, or lifecycle questions, which have dedicated skills. If the underlying feature flag itself returns wrong values rather than the results being skewed, hand off to feature flag debugging instead.

Related Skills

Explore More Data & Analytics Skills→
wattdatawattdata
5

help-ticket

Draft and file Watt support tickets after explicit confirmation.

Official
Advanced
PostHogPostHog
58

managing-experiment-lifecycle

Guide experiment lifecycle transitions with preconditions and decision criteria.

Official
Advanced
PostHogPostHog
58

feature-flags-php

Implement and evaluate PostHog feature flags in PHP applications.

Official
Advanced
xtrm-devxtrm-dev
4

data-analyst

Analyze datasets using SQL, pandas, and statistical methods.

Official
Intermediate