One-click install
npx skills add https://github.com/FrancisVarga/the-dream-machine --skill test-flakiness-francisvarga
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: test-flakiness
Source: https://github.com/FrancisVarga/the-dream-machine/tree/main/.claude/skills/test-flakiness
Command: npx skills add https://github.com/FrancisVarga/the-dream-machine --skill test-flakiness-francisvarga

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill reduces wasted engineering time caused by non-deterministic (flaky) tests that intermittently fail and undermine trust in CI results.

Core Features & Use Cases

  • Flakiness detection from test history: Reads CI run logs or existing regression-suite quarantine sections to determine which tests fluctuate between pass and fail.
  • Root-cause classification: Identifies likely sources such as timing/async issues, order dependency, random seed usage, resource leaks, external state, floating-point comparisons, and scene/prefab load races.
  • Actionable remediation: Recommends whether to quarantine immediately, investigate soon, or monitor, and prepares optional flakiness reports while updating the quarantine registry.

Quick Start

Run /test-flakiness scan to analyze available CI logs and update tests/regression-suite.md quarantine for detected flaky tests.

Frequently Asked Questions about test-flakiness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I detect flaky tests from JUnit XML logs and CI run history?

Flaky test detection aggregates pass/fail outcomes across multiple CI runs to identify intermittent failures without code changes. It reads JUnit XML or plain-text CI logs, builds per-test result history, and classifies likely failure causes like timing issues or order dependencies.

Why do intermittent test failures keep happening even when no code changes were made?

Intermittent failures without code changes indicate flaky tests caused by timing or async issues, order dependency, random seed usage, resource leaks, external state, floating-point comparisons, or scene and prefab load races. Analyzing CI log history isolates these non-deterministic behaviors.

Can I use this approach to scan existing test result artifacts in bulk?

Yes, bulk scanning of available test result artifacts is supported. It reads existing CI logs and regression-suite quarantine sections to determine which tests fluctuate between pass and fail, then updates quarantine report files with user approval.

What's the best way to quarantine flaky tests identified during CI log analysis?

The best way to quarantine flaky tests is to classify them by root cause, recommend whether to quarantine immediately, investigate soon, or monitor, prepare a flakiness report, and update the quarantine registry with explicit user approval.

Does this method work with Jest JUnit XML output for regression testing?

Yes, it processes Jest JUnit XML output and plain-text CI logs for regression testing. It builds per-test result history from these formats to identify intermittent failures and classify likely causes like timing issues or resource leaks.

What are the limitations of identifying intermittent failures from CI log history?

Detection relies on having sufficient CI run history and accessible test result artifacts. It classifies likely failure causes and recommends remediation, but updating quarantine report files and applying fixes requires explicit user approval and intervention.