test-flakiness

Analyze CI logs and JUnit XML to detect flaky tests and recommend quarantine actions.

Updated Apr 4, 2026

Search Tags:#root cause analysis #test reliability #regression suite #quarantine #junit xml #test flakiness #ci logs

Ready to run

npx skills add https://github.com/WindXRan/y3td --skill test-flakiness-windxran

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: test-flakiness
Source: https://github.com/WindXRan/y3td/tree/main/.claude/skills/test-flakiness
Command: npx skills add https://github.com/WindXRan/y3td --skill test-flakiness-windxran

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Flaky tests intermittently fail or pass without code changes, eroding trust in CI results and masking real regressions.

Core Features & Use Cases

CI log and test result parsing: Detects pass/fail variability by reading JUnit-style XML or plain-text CI logs across multiple runs.
Flakiness classification and root-cause hints: Aggregates fail-rate history per test and maps symptoms to likely causes such as timing/async, order dependency, RNG, or resource leaks.
Actionable remediation and reporting: Recommends whether to quarantine immediately, investigate soon, or monitor; updates a quarantine section in the regression suite and can generate an external flakiness report with per-test guidance.

Quick Start

Run /test-flakiness scan to scan available CI logs, identify flaky tests, and produce recommendations for quarantine or fixes.

Frequently Asked Questions about test-flakiness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I detect flaky tests from CI logs and JUnit XML results?▼

Flaky tests are identified by parsing JUnit XML or plain-text CI logs across multiple runs to aggregate pass/fail outcomes per test and flag inconsistent results without code changes.

Why does my regression suite fail non-reproducibly in CI?▼

Non-reproducible red CI often stems from flaky tests with intermittent pass/fail variability; analyzing historical CI run logs and quarantine records isolates tests with inconsistent outcomes for statistical aggregation and remediation.

Can I classify flaky test root causes like timing issues or resource leaks?▼

Yes, flakiness classification maps aggregated fail-rate symptoms to likely root causes including timing or async issues, order dependency, RNG, or resource leaks to guide targeted fixes.

What is the best way to quarantine flaky tests before a release?▼

The best approach evaluates statistical pass-rate aggregation and root-cause classification to recommend immediate quarantine, further investigation, or monitoring, then updates the regression suite's quarantine section accordingly.

Does this test flakiness analysis work with plain-text CI logs or only JUnit XML?▼

Test flakiness analysis supports both JUnit-style XML and plain-text CI logs, parsing either format across multiple runs to build per-test outcome history and generate actionable flakiness reports.