test-flakiness

Detect flaky tests by analyzing CI logs and test result history across runs.

13|Updated Jul 30, 2026
One-click install
npx skills add https://github.com/frabcd/codex-ai-game-studio --skill test-flakiness-frabcd
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: test-flakiness
Source: https://github.com/frabcd/codex-ai-game-studio/tree/main/plugins/ai-game-studio/skills/test-flakiness
Command: npx skills add https://github.com/frabcd/codex-ai-game-studio --skill test-flakiness-frabcd

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Flaky tests that intermittently pass and fail erode trust in CI pipelines, causing teams to ignore genuine failures. This Skill identifies non-deterministic tests from run history, classifies likely root causes, and recommends whether to quarantine or fix each one. ## Core Features & Use Cases - CI Log Parsing: Reads JUnit XML results (GdUnit4, Unity game-ci) and plain-text logs from Godot, Unity, and Unreal test runners to build per-test pass/fail history. - Flakiness Classification: Categorizes tests by failure rate (high, moderate, suspected) and diagnoses likely causes such as timing/async issues, order dependency, random seeds, resource leaks, and floating-point comparisons. - Quarantine Management: Updates the quarantine section of tests/regression-suite.md and optionally writes a full flakiness report to production/qa/, always with explicit user approval. - Use Case: After several CI runs where developers dismiss failures as "probably flaky", run this Skill to statistically confirm which tests are non-deterministic and get engine-specific fix guidance. ## Quick Start Ask the AI to run the test-flakiness skill in scan mode to analyze all available CI logs and report which tests are flaky with recommended actions.

Frequently Asked Questions about test-flakiness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I detect flaky tests from CI logs?

Run the skill in scan mode to parse all available CI logs in .github/ or test-results/ directories. It builds a pass/fail history per test across runs and flags any test showing both outcomes without code changes as flaky.

How to find flaky tests in Godot or Unity projects?

For Godot, the skill reads GdUnit4 JUnit XML output from test-results/. For Unity, it parses NUnit XML from game-ci test runners. Unreal automation logs in Saved/Logs/ are scanned for Result: Success and Result: Fail patterns.

What failure rate makes a test flaky enough to quarantine?

Tests failing in more than 25% of runs are high flakiness and should be quarantined immediately. Tests failing 5-25% should be investigated and fixed, while 1-5% failure rates are marked as suspected and monitored.

Does this skill delete or modify my test files?

No, the skill never deletes test files. Quarantine means adding skip annotations and listing tests in the regression suite registry. All writes to regression-suite.md or report files require explicit user approval first.

Why does flakiness detection need multiple CI runs?

Statistical confidence requires run history; with fewer than 3 runs, findings are flagged as suspected rather than confirmed. A single pass/fail pair cannot distinguish genuine flakiness from a one-off environment issue.