test-flakiness

Detects flaky tests by analyzing CI logs and test result history across runs.

Updated Aug 2, 2026
One-click install
npx skills add https://github.com/claretagrapelike32/codex-ai-game-studio --skill test-flakiness-claretagrapelike32
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: test-flakiness
Source: https://github.com/claretagrapelike32/codex-ai-game-studio/tree/main/plugins/ai-game-studio/skills/test-flakiness
Command: npx skills add https://github.com/claretagrapelike32/codex-ai-game-studio --skill test-flakiness-claretagrapelike32

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Flaky tests that pass and fail intermittently erode trust in CI pipelines, causing teams to ignore genuine failures. This Skill identifies non-deterministic tests from CI run logs, classifies their likely root causes, and recommends whether to quarantine or fix each one. ## Core Features & Use Cases - CI Log Parsing: Reads JUnit/NUnit XML results and plain-text logs from Godot (GdUnit4), Unity, and Unreal test runners to build per-test pass/fail history. - Flakiness Classification: Categorizes tests by failure rate (high, moderate, suspected) and diagnoses likely causes such as timing issues, order dependency, random seeds, or floating-point comparisons. - Quarantine Management: Updates the quarantine section of tests/regression-suite.md and optionally writes a full flakiness report to production/qa/. - Use Case: After several CI runs, a team notices intermittent red builds. Run this Skill to scan test-results/ logs, identify that a physics test fails 30% of the time due to a float comparison, and get a concrete fix recommendation using epsilon comparison. ## Quick Start Ask the AI to run the test-flakiness skill in scan mode to analyze all available CI logs and report which tests are flaky with recommended actions.

Frequently Asked Questions about test-flakiness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I detect flaky tests from CI logs?

Run the skill in scan mode to parse all available CI logs in .github/ or test-results/ directories. It builds a pass/fail history per test across runs and flags any test showing both outcomes without code changes as flaky.

What test result formats does flaky test detection support?

It supports JUnit XML output from Godot GdUnit4 and Unity game-ci test runners, plus plain-text logs from Unreal automation using Result: Success/Fail patterns. You can also pass a specific log file path as an argument.

When should I quarantine a flaky test instead of fixing it?

Quarantine tests failing in more than 25% of runs immediately using engine-specific skip annotations. Tests failing 5-25% should be investigated and fixed directly, while tests under 5% failure rate should only be monitored as suspected flaky.

Why does my test fail intermittently with no code changes?

Common causes include timing/async issues, test order dependency, unseeded randomness, resource leaks, external state, and floating-point equality comparisons. The skill inspects the test file for these patterns to classify the likely cause and suggest a fix.

How many CI runs are needed for reliable flakiness detection?

At least 3 runs are needed for meaningful signal, and 5 or more runs provide better statistical confidence. With fewer than 3 runs, findings are flagged as suspected rather than confirmed flaky.