skills.rest
easonlao
easonlaoCOMMUNITY
Explore all skills by this author →
GitHub

test-flakiness

Analyze CI test logs to classify flaky tests and recommend quarantine actions.

Updated Apr 25, 2026
Search Tags:#root cause analysis#regression testing#test reliability#quarantine#junit xml#test flakiness#ci logs
One-click install
npx skills add https://github.com/easonlao/jiazi-game --skill test-flakiness-easonlao
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: test-flakiness
Source: https://github.com/easonlao/jiazi-game/tree/main/.claude/skills/test-flakiness
Command: npx skills add https://github.com/easonlao/jiazi-game --skill test-flakiness-easonlao

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Detects non-deterministic (flaky) tests that intermittently pass or fail without code changes, helping prevent teams from silently ignoring unreliable CI results.

Core Features & Use Cases

  • CI log and result history analysis: Parses JUnit-style XML and plain-text CI outputs to reconstruct per-test pass/fail history across runs.
  • Flakiness classification and root-cause hints: Identifies likely causes such as timing/async issues, order dependency, random seed problems, resource leaks, and floating-point comparison errors.
  • Actionable remediation workflow: Updates a quarantine section in tests/regression-suite.md and can generate an additional flakiness report for deeper diagnosis and fix guidance.

Use case: After multiple CI runs start showing red intermittently, run this skill to produce a ranked list of flaky tests with fail rates and recommended quarantine or fix paths.

Quick Start

Run test-flakiness in scan mode by saying: scan the available CI test logs and update tests/regression-suite.md with any flaky tests found.

Frequently Asked Questions about test-flakiness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I identify flaky tests from CI logs?▼

Flaky tests are identified by parsing CI logs and JUnit XML outputs to aggregate pass/fail outcomes per test identifier across multiple runs. This measures pass/fail variability to detect non-deterministic test behavior.

How do I find the root cause of test flakiness in a regression suite?▼

Root cause analysis of test flakiness examines parsed failure patterns from regression suite logs to classify likely causes. It identifies timing issues, order dependency, random seed problems, resource leaks, and floating-point comparison errors.

Does this flakiness detection work with JUnit XML and plain-text CI outputs?▼

Yes, flakiness detection supports both JUnit XML and plain-text CI outputs. It parses these formats to reconstruct per-test pass/fail history across runs, targeting regression suites where intermittent failures occur.

What is the best way to quarantine flaky tests?▼

The best way to quarantine flaky tests is to generate a ranked list with fail rates and update the quarantine section in tests/regression-suite.md. This actionable remediation workflow prevents teams from silently ignoring unreliable CI results.

Can I measure test pass/fail variability over time without code changes?▼

Yes, you can measure test pass/fail variability over time by analyzing existing CI test result logs or quarantine history. This detects non-deterministic tests that intermittently pass or fail without code changes.

Related Skills

Explore More Software Engineering Skills→
XGanQAQXGanQAQ
1

test-flakiness

Analyze CI test logs to detect intermittent PASS/FAIL flaky tests.

Community
Advanced
razorpayrazorpay
31

pr-autopilot

Diagnose CI failures and propose fixes for GitHub pull requests.

Official
Advanced
alangebalangeb

test-suite-monitor

Run and monitor test suites in background tmux sessions with comparative analysis.

Community
Intermediate
jmaglyjmagly
175

flaky-detect

Analyze CI history and test execution patterns to identify flaky tests.

Community
Advanced