test-flakiness

Name: test-flakiness
Availability: InStock
Author: solace

Analyze CI logs and JUnit/XML results to detect flaky tests.

Updated Apr 26, 2026

Search Tags:#remediation #regression #flaky-tests #quarantine #test-results #ci-logs

Ready to run

npx skills add https://github.com/solace-ill/vector-flow --skill test-flakiness-solace-ill

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: test-flakiness
Source: https://github.com/solace-ill/vector-flow/tree/main/.claude/skills/test-flakiness
Command: npx skills add https://github.com/solace-ill/vector-flow --skill test-flakiness-solace-ill

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Detect flaky tests by analyzing CI run logs and test results to identify tests that pass and fail inconsistently. This helps teams avoid chasing intermittent issues and maintain CI reliability.

Core Features & Use Cases

Flakiness detection: aggregates per-test pass/fail history across runs to highlight intermittently failing tests.
Remediation guidance: provides recommended actions (quarantine, investigate, monitor) and tracks changes in a regression suite.
Cross-project applicability: works with common CI setups and various game/engine ecosystems to surface flaky tests.

Quick Start

Run a scan of available CI logs to identify flaky tests and generate a remediation plan.

Frequently Asked Questions about test-flakiness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I detect flaky tests from CI logs and test results?▼

To detect flaky tests from CI logs, you can parse JUnit/XML results or plain text logs to build per-test pass/fail histories. This process aggregates run data to highlight intermittently failing tests across your CI pipeline.

What is the best way to quarantine flaky tests and guide remediation?▼

Quarantining flaky tests involves analyzing per-test histories to compute fail rates and classify flakiness levels. This classification generates actionable remediation guidance, recommending whether to quarantine, investigate, or monitor the affected tests.

Does flaky test detection work with plain text CI logs or do I need JUnit XML?▼

Flaky test detection works with both plain text CI logs and JUnit/XML formats. You can parse either format to extract test results, build per-test pass/fail histories, and compute fail rates across multiple CI runs.

Can I analyze test results across multiple CI runs and different project environments?▼

Yes, you can analyze test results across multiple CI runs and various project environments. The detection process applies cross-project to surface flaky tests by tracking inconsistencies in pass/fail histories regardless of the specific game or engine ecosystem.

How do I compute fail rates to classify flakiness levels in a regression suite?▼

To compute fail rates and classify flakiness levels, the system calculates the ratio of failures to total executions from your per-test histories. This data tracks changes in your regression suite and triggers actionable remediation guidance.