test-flakiness

Analyze CI run logs and test histories to detect flaky tests and maintain a registry.

Updated Apr 24, 2026

Search Tags:#automation #testing #logs #quality #regression #ci #flaky

Ready to run

npx skills add https://github.com/rakhnianskii9/God-Godot --skill test-flakiness-rakhnianskii9

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: test-flakiness
Source: https://github.com/rakhnianskii9/God-Godot/tree/main/.github/skills/test-flakiness
Command: npx skills add https://github.com/rakhnianskii9/God-Godot --skill test-flakiness-rakhnianskii9

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Detects non-deterministic (flaky) tests by analyzing CI logs and test histories, identifies intermittent failures, and maintains a registry to guide quarantine or fixes.

Core Features & Use Cases

Flaky test detection across CI runs with per-test pass rate aggregation
Quarantine guidance and remediation planning for flaky tests
Automated updates to tests/regression-suite.md and optional flakiness reports

Quick Start

Run the tool against a CI log to identify flaky tests and add them to the registry for triage.

Frequently Asked Questions about test-flakiness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I identify flaky tests across multiple CI runs?▼

To identify flaky tests across CI runs, analyze CI logs and test-result history to detect intermittent failures and aggregate pass rates per test. This process highlights non-deterministic tests for triage.

What is a flaky test registry and how does it help with CI quality?▼

A flaky test registry tracks non-deterministic tests identified in CI logs to maintain quality. It provides quarantine guidance and remediation planning by aggregating pass rates, helping isolate intermittent failures from blocking pipelines.

Can I scan a single CI log file to detect intermittent test failures?▼

Yes, you can analyze a single CI log file to detect intermittent test failures. The tool supports scanning individual logs or all available histories to identify non-deterministic behavior and update the registry.

How do I quarantine flaky tests in a regression suite?▼

To quarantine flaky tests in a regression suite, use the generated flakiness reports and registry to guide remediation. The tool automatically updates regression-suite files to isolate non-deterministic tests from blocking CI pipelines.

Does flaky test detection work with Godot-like CI pipelines?▼

Yes, flaky test detection is designed to apply to Godot-like CI pipelines. It analyzes test-result history across multiple runs to detect intermittent failures and maintain a registry for quarantine or fix guidance.