flaky-test-diagnoser

To diagnose intermittent test failures, you need to measure the flakiness rate through repeated test execution and perform root cause analysis by evaluating test isolation, shared state, and timing to distinguish competing hypotheses. This Skill automates that diagnostic process by running repeated executions, generating hypotheses about ordering or environment factors, and recording structured evidence to isolate the non-deterministic root cause.

Flaky tests are caused by shared state, test ordering dependencies, timing issues, resource leaks, and non-deterministic inputs. Root cause analysis identifies them by running isolation, bisection, and distinguishing experiments to narrow competing hypotheses and produce structured evidence of the specific environmental factor.

Yes, you can use hypothesis testing to isolate non-deterministic test failures across common test runners and programming languages. The Skill applies isolation and timing analysis to distinguish experiments, measuring flakiness to find reproducible evidence regardless of the specific environment.

The best way to measure the flakiness rate of randomly failing tests is to execute the specified test repeatedly to establish statistical pass and failure rates. This repeated execution provides the baseline measurement needed to conduct hypothesis generation and timing analysis.

Yes, test isolation analysis works for environment-sensitive tests by evaluating external dependencies, shared state, and resource leaks. It uses bisection and distinguishing experiments to narrow competing hypotheses and produce a reproducible diagnosis of the environmental factors.