skills.rest
imgamer
imgamerCOMMUNITY
Explore all skills by this author →
GitHub

test-flakiness

Analyze CI logs and JUnit XML to detect intermittent test failures.

Updated Dec 22, 2024
Search Tags:#root cause analysis#test reliability#regression suite#junit xml#test flakiness#ci logs#quarantine tests
One-click install
npx skills add https://github.com/imgamer/AILearn --skill test-flakiness-imgamer
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: test-flakiness
Source: https://github.com/imgamer/AILearn/tree/main/game-studios/Claude-Code-Game-Studios-e375f/.claude/skills/test-flakiness
Command: npx skills add https://github.com/imgamer/AILearn --skill test-flakiness-imgamer

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill identifies non-deterministic (flaky) tests that sometimes pass and sometimes fail without code changes, preventing teams from ignoring red CI runs that mask real regressions.

Core Features & Use Cases

  • Parse CI test history: Reads CI run logs or test result artifacts (JUnit XML or plain text) to build per-test pass/fail histories.
  • Detect and classify flakiness: Flags tests that show intermittent PASS/FAIL patterns and estimates likely root causes such as timing issues, order dependency, random seeds, resource leaks, external state, floating-point comparisons, or scene/prefab load races.
  • Recommend remediation: Guides whether to quarantine immediately or fix soon, then supports updating the regression suite quarantine section and producing an optional flakiness report.

Quick Start

Run the command to scan available CI logs and update the quarantine recommendations in tests/regression-suite.md.

Frequently Asked Questions about test-flakiness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I detect flaky tests from JUnit XML or CI logs?▼

Detect flaky tests by scanning CI run logs or JUnit XML test result history to find intermittent PASS/FAIL outcomes for the same test across multiple runs. The Skill parses test identifiers and outcomes to build per-test pass/fail histories for analysis.

What causes intermittent test failures in a regression suite?▼

Intermittent test failures are often caused by timing issues, order dependency, random seeds, resource leaks, external state, floating-point comparisons, or scene and prefab load races. Analyzing CI test history helps classify these likely root causes.

How do I quarantine flaky tests before they hide real regressions?▼

Quarantine flaky tests by scanning test result artifacts to flag tests with intermittent PASS/FAIL patterns, then updating the regression suite quarantine section. This prevents non-deterministic tests from masking real CI regressions.

Can I analyze plain-text CI output to find non-deterministic tests?▼

Yes, you can analyze plain-text CI output alongside JUnit XML files. The Skill parses both formats to extract test identifiers and outcomes across multiple CI runs, building the pass/fail histories needed to detect non-deterministic behavior.

When do I need to analyze test flakiness in my CI pipeline?▼

Analyze test flakiness during polish phases when statistical signal improves across multiple CI runs. This is essential when non-deterministic PASS/FAIL outcomes cause teams to ignore red CI runs that mask real regressions.

What is the best way to classify root causes of flaky tests?▼

The best way to classify root causes of flaky tests is to parse CI test history and analyze intermittent PASS/FAIL patterns. This approach estimates likely causes like timing issues, order dependency, random seeds, or resource leaks to guide remediation.

Related Skills

Explore More Software Engineering Skills→
XGanQAQXGanQAQ
1

test-flakiness

Analyze CI test logs to detect intermittent PASS/FAIL flaky tests.

Community
Advanced
emcconnellemcconnell

test-flakiness

Aggregates CI pass/fail histories to identify intermittent test failures across Godot, Unity, Unreal projects.

Community
Intermediate
scomofoscomofo

test-flakiness

Analyze CI logs to detect flaky tests and classify root causes.

Community
Intermediate
sumilee-pcusumilee-pcu

test-flakiness

Detect non-deterministic test failures by analyzing CI logs and test artifacts.

Community
Advanced