skills.rest
XGanQAQ
XGanQAQCOMMUNITY
Explore all skills by this author →
GitHub

test-flakiness

Analyze CI test logs to detect intermittent PASS/FAIL flaky tests.

1|Updated May 5, 2026
Search Tags:#github actions#test flakiness#ci logs#test quarantine#qa regression#intermittent failures#nunit junit xml
One-click install
npx skills add https://github.com/XGanQAQ/Unity-Claude-Code-Game-Studios --skill test-flakiness-xganqaq
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: test-flakiness
Source: https://github.com/XGanQAQ/Unity-Claude-Code-Game-Studios/tree/main/.claude/skills/test-flakiness
Command: npx skills add https://github.com/XGanQAQ/Unity-Claude-Code-Game-Studios --skill test-flakiness-xganqaq

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill identifies flaky (non-deterministic) tests that intermittently pass or fail without code changes, reducing the temptation to ignore red CI runs and helping teams restore trust in automated testing.

Core Features & Use Cases

  • CI log and history parsing: Reads GitHub Actions test artifacts (JUnit/NUnit XML or plain-text logs) or a provided log file to build a pass/fail history per test.
  • Flakiness detection with thresholds: Classifies flakiness severity (high/moderate/low) based on observed fail rates across multiple runs.
  • Actionable remediation: Recommends whether to quarantine immediately or investigate/fix next, and supports updating the regression suite quarantine registry plus an optional detailed flakiness report.
  • Likely-cause classification: Detects likely categories such as timing/async issues, order dependency, random seed usage, resource leaks, external state, floating-point comparison problems, and scene/prefab load races.
  • Use Case: After several CI runs where the same Unity/Godot/Unreal test alternates between pass and fail, use this Skill to identify the most suspect tests, quarantine the high flakiness ones, and surface the likely root causes so you can fix them.

Quick Start

Run test-flakiness in scan mode to analyze available CI test logs and update tests/regression-suite.md with any newly detected flaky tests.

Frequently Asked Questions about test-flakiness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I detect flaky tests from GitHub Actions CI logs?▼

Flaky test detection from GitHub Actions CI logs works by parsing JUnit/NUnit XML artifacts or plain-text logs across multiple runs to find intermittent PASS/FAIL patterns. It computes fail rates with specific thresholds to classify flakiness severity without requiring code changes.

What causes intermittent test failures in Unity, Godot, and Unreal projects?▼

Intermittent test failures in game engines are often caused by timing/async issues, order dependency, random seed usage, resource leaks, external state, floating-point comparison problems, and scene/prefab load races. Analyzing test result history helps classify these likely causes.

Can I parse NUnit and JUnit XML artifacts to find intermittent PASS/FAIL patterns?▼

Yes, parsing NUnit and JUnit XML artifacts allows you to extract test identifiers and outcomes to build a pass/fail history per test. This history is used to compute fail rates and identify intermittent PASS/FAIL patterns across multiple CI runs.

How do I quarantine flaky tests in a regression suite?▼

To quarantine flaky tests in a regression suite, the system recommends immediate quarantine for high flakiness items and investigation for lower severity ones. It can optionally update the tests/regression-suite.md quarantine registry and generate a dated QA report.

What is the best way to analyze CI test logs for non-deterministic test failures?▼

The best way to analyze CI test logs for non-deterministic test failures is to scan available test artifacts across multiple runs, compute fail rates against flakiness thresholds, and classify likely causes. This approach identifies suspect tests and recommends whether to quarantine or fix them.

Do I need code changes to identify flaky tests from test result sources?▼

No, you do not need code changes to identify flaky tests from test result sources. The detection process relies entirely on analyzing existing CI test logs or regression-suite quarantine history to find intermittent PASS/FAIL patterns.

Related Skills

Explore More Software Engineering Skills→
LordKuperLordKuper

test-flakiness

Detects flaky tests by aggregating CI logs and analyzing pass/fail patterns across runs.

Community
Advanced
scomofoscomofo

test-flakiness

Analyze CI logs to detect flaky tests and classify root causes.

Community
Intermediate
polsegupolsegu

test-flakiness

Detect nondeterministic failures by parsing CI logs and aggregating per-test pass/fail histories.

Community
Intermediate
rizaldiemrizaldiem

unit-test-scheduled-async

Unit-test Spring @Async and @Scheduled methods with CompletableFuture and Awaitility.

Community
Intermediate