test-flakiness

Analyze CI logs and test history to detect flaky tests.

72|7|Updated Apr 29, 2026

Search Tags:#regression testing #test reliability #quarantine #junit xml #test flakiness #qa triage #ci logs

Ready to run

npx skills add https://github.com/striderZA/OpenCodeGameStudios --skill test-flakiness-striderza

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: test-flakiness
Source: https://github.com/striderZA/OpenCodeGameStudios/tree/main/.opencode/skills/test-flakiness
Command: npx skills add https://github.com/striderZA/OpenCodeGameStudios --skill test-flakiness-striderza

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill addresses the chaos of flaky (non-deterministic) tests that intermittently fail and waste time during CI, obscuring real regressions.

Core Features & Use Cases

Flakiness detection from CI history: Scans or parses CI logs/results to find tests that both pass and fail across runs.
Fail-rate classification: Categorizes tests into high, moderate, or low flakiness based on observed failure frequency.
Actionable remediation guidance: Recommends quarantine for high flakiness or targeted investigation/fixes for lower tiers, and can generate a dated flakiness report.
Use case: Your CI has recurring red builds for the same test name, and your team suspects it is unreliable—use this Skill to verify it statistically and decide whether to quarantine or fix.

Quick Start

Ask the AI to run test flakiness analysis by scanning available CI logs and updating the quarantine section of tests/regression-suite.md with the findings.

Frequently Asked Questions about test-flakiness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I detect flaky tests from CI logs and JUnit XML history?▼

Flaky tests are detected by parsing CI logs and JUnit XML to aggregate per-test outcomes across runs, identifying inconsistent pass/fail results. This analysis reveals tests with non-deterministic behavior caused by timing issues or resource leaks.

What causes non-deterministic test failures in a regression suite?▼

Non-deterministic test failures stem from timing dependencies, test order issues, random seeds, resource leaks, external state, floating-point comparisons, or scene load races. Analyzing test result history isolates these intermittent issues from real regressions.

When should I quarantine a flaky test versus fixing it?▼

Quarantine is recommended for tests with high flakiness fail-rates to stop recurring red builds, while moderate or low flakiness suggests targeted investigation and fixes. This classification prevents flaky tests from obscuring real regressions.

Can I automatically update regression-suite.md with quarantined tests?▼

Yes, the analysis can optionally update the quarantine section of tests/regression-suite.md with the findings. It also generates a dated flakiness report detailing the classified fail-rates and recommended next steps.

Does test flakiness analysis work with plain-text CI logs?▼

Yes, detecting test flakiness works by parsing both JUnit XML and plain-text CI logs. This allows it to aggregate run outcomes and identify intermittent failures even without structured XML test reports.