What problem does it solve?
Flaky tests cause development teams to dismiss legitimate CI failures as intermittent issues, masking genuine bugs and reducing trust in automated testing pipelines. This Skill automates the detection of non-deterministic tests, classifies their root causes, and provides clear recommendations to either quarantine or fix them, restoring CI reliability.
Core Features & Use Cases
- CI Log Analysis: Parses GitHub Actions logs, JUnit XML test results (for Godot, Unity, Unreal engines), and plain text CI output to aggregate pass/fail rates per test across multiple runs.
- Flakiness Classification: Identifies likely root causes including timing/async issues, order dependencies, random seed usage, resource leaks, and floating point comparison errors.
- Actionable Recommendations: Suggests targeted fixes for moderate flakiness, immediate quarantine steps for high flakiness, and monitoring guidance for suspected flaky tests.
- Registry Maintenance: Updates the project's
tests/regression-suite.md quarantine section and can generate full flakiness reports for team review.
Use Case: For a game development team running CI pipelines for a Godot project, use this Skill after multiple test runs to identify flaky GdUnit4 tests, get specific fix instructions (like adding explicit awaits or using epsilon comparisons), and update the regression suite to track quarantined tests.
Quick Start
Use the test-flakiness skill to scan your latest CI run logs and receive a prioritized list of flaky tests with recommended fixes and quarantine steps.
