test-flakiness

Analyze JUnit XML and CI logs to classify flaky tests.

Updated Aug 23, 2026

Search Tags:#junit #ci #test-automation #flaky-tests #regression-testing #quarantine #test-analysis

Ready to run

npx skills add https://github.com/nevesb/idol-agency-maker --skill test-flakiness-nevesb

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: test-flakiness
Source: https://github.com/nevesb/idol-agency-maker/tree/main/.claude/skills/test-flakiness
Command: npx skills add https://github.com/nevesb/idol-agency-maker --skill test-flakiness-nevesb

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Detect intermittent, non-deterministic test failures across CI runs and surface reliable guidance so teams stop ignoring red builds and can prioritize fixes or quarantine appropriately.

Core Features & Use Cases

CI Log Parsing: Ingest JUnit XML and plain-text CI logs from common locations such as test-results/ or .github/ to build a per-test run history.
Flakiness Detection & Classification: Aggregate pass/fail histories, compute fail-rate thresholds, and classify causes like timing/async, order dependency, random seeds, resource leaks, external state, float comparisons, or scene load races.
Remediation & Reporting: Recommend quarantine, investigation, or monitoring actions; generate an in-conversation summary; optionally append quarantined tests to tests/regression-suite.md and write production/qa/flakiness-report-[date].md after explicit approval.
Collaboration Safety: Preserve existing quarantine entries, require user consent before edits or file writes, and flag low-confidence findings when run history is insufficient.

Quick Start

Run /test-flakiness scan to analyze available CI logs and return a flakiness report with recommendations.

Frequently Asked Questions about test-flakiness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I detect flaky tests from JUnit XML and CI logs?▼

Flaky test detection aggregates pass/fail histories from JUnit XML and plain CI logs across multiple runs. It computes fail-rate thresholds and classifies causes like timing issues, order dependencies, or random seeds to surface non-deterministic failures.

What causes non-deterministic test failures in CI pipelines?▼

Non-deterministic test failures stem from timing or async races, order dependencies, random seeds, resource leaks, external state, float comparisons, or scene load races. Analyzing CI run artifacts helps classify these specific flakiness causes.

Can I parse test results from Godot, Unity, and Unreal engines to find flaky tests?▼

Yes, flakiness analysis supports game engines like Godot, Unity, and Unreal. It parses their CI run artifacts and JUnit XML outputs to compute per-test fail rates and classify intermittent failures.

How do I quarantine flaky tests in my regression suite?▼

To quarantine flaky tests, the analysis recommends quarantine actions based on computed fail-rate thresholds. After explicit approval, it appends quarantined tests to tests/regression-suite.md while preserving existing quarantine entries.

What's the best way to generate a flakiness report for CI test runs?▼

Generating a flakiness report involves analyzing multiple CI run artifacts to aggregate pass/fail histories. After explicit approval, it writes a dated report to production/qa/flakiness-report-[date].md with remediation recommendations.

Does flaky test detection work with insufficient CI run history?▼

When run history is insufficient, the analysis flags low-confidence findings rather than guessing. It requires adequate pass/fail artifacts to compute reliable fail-rate thresholds and accurately classify flaky tests.