test-flakiness

Detect nondeterministic failures by parsing CI logs and aggregating per-test pass/fail histories.

Updated Apr 10, 2026
One-click install
npx skills add https://github.com/polsegu/Claude-Code-Game-Dev --skill test-flakiness-polsegu
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: test-flakiness
Source: https://github.com/polsegu/Claude-Code-Game-Dev/tree/main/.claude/skills/test-flakiness
Command: npx skills add https://github.com/polsegu/Claude-Code-Game-Dev --skill test-flakiness-polsegu

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Flaky tests produce intermittent pass/fail results that erode trust in CI, waste developer time, and mask real regressions; this Skill finds those nondeterministic failures and helps teams decide whether to quarantine or fix them. It reduces noise in pipelines by surfacing tests with mixed outcomes across runs, explaining likely causes, and recommending concrete remediation steps.

Core Features & Use Cases

  • Log parsing and aggregation: Parse JUnit XML and plain-text CI logs to build per-test pass/fail histories across multiple runs.
  • Flakiness classification: Compute fail-rate thresholds (high/moderate/low) and suggest likely causes such as timing issues, order dependency, random seeds, resource leaks, external state, or floating-point comparisons.
  • Actionable recommendations and reporting: Recommend quarantine, fixes, or monitoring; optionally append quarantined tests to tests/regression-suite.md and write a full flakiness report to production/qa.
  • Use case: Run during the polish phase or after multiple CI runs when developers start dismissing failures as "flaky," to produce a prioritized list of tests to quarantine or fix.

Quick Start

Run /test-flakiness scan to analyze available CI logs, detect flaky tests, and propose quarantine or fixes.

Frequently Asked Questions about test-flakiness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I detect flaky tests in my CI logs?

To detect flaky tests in CI logs, you can parse JUnit XML and plain-text test logs to build per-test pass/fail histories across multiple runs, then compute fail-rate thresholds to identify intermittent failures. The Skill aggregates these histories to surface nondeterministic results.

What causes flaky tests in JUnit test suites?

Flaky tests in JUnit suites are commonly caused by timing issues, order dependency, random seeds, resource leaks, external state, or floating-point comparisons. The Skill classifies likely causes by analyzing pass/fail histories across repeated CI runs.

Can I analyze GitHub Actions test results for intermittent failures?

Yes, you can analyze GitHub Actions test results for intermittent failures by applying the Skill to repositories with repeated CI runs. It parses test-results directories and saved local logs to identify nondeterministic failures and classify their causes.

When should I quarantine flaky tests instead of fixing them?

You should quarantine flaky tests instead of fixing them when you need to reduce noise in pipelines during the polish phase. The Skill recommends quarantine, fixes, or monitoring based on computed fail-rate thresholds and optionally appends quarantined tests to regression-suite.md.

Does test-flakiness work without any external dependencies?

Yes, test-flakiness works without external dependencies because it operates directly on existing CI run logs, JUnit XML files, and plain-text test outputs to compute fail-rate thresholds and generate a flakiness report.