skills.rest
kay6198835
kay6198835COMMUNITY
Explore all skills by this author →
GitHub

test-flakiness

Analyze CI run logs to identify flaky tests and recommend quarantine actions.

Updated Feb 8, 2024
Search Tags:#automation#flaky-tests#quarantine#test-results#ci-logs#regression-suite
One-click install
npx skills add https://github.com/kay6198835/Game-RPG --skill test-flakiness-kay6198835
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: test-flakiness
Source: https://github.com/kay6198835/Game-RPG/tree/main/.claude/skills/test-flakiness
Command: npx skills add https://github.com/kay6198835/Game-RPG --skill test-flakiness-kay6198835

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Detect non-deterministic (flaky) tests by reading CI run logs or test result history. Aggregates pass rates per test, identifies intermittent failures, recommends quarantine or fix, and maintains a flaky test registry. Best run during Polish phase or after multiple CI runs.

Core Features & Use Cases

  • Parse CI logs across GitHub Actions, GitLab CI, and local runs to extract test results.
  • Identify flaky tests based on pass/fail history and provide remediation guidance.
  • Update the regression suite with quarantine annotations and generate optional reports.

Quick Start

Run the skill with a path to CI logs or use scan to analyze available logs and generate a flakiness report.

Frequently Asked Questions about test-flakiness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I identify flaky tests from CI logs?▼

Detect flaky tests by parsing CI logs to extract test results and compute per-test pass/fail trajectories across multiple runs. The analysis identifies intermittent failures occurring without code changes and categorizes test flakiness by severity.

Does this test flakiness detection work with GitHub Actions and GitLab CI?▼

Yes, test flakiness detection supports parsing CI run logs across GitHub Actions, GitLab CI, and local runs to extract test results and aggregate pass rates per test for intermittent failure identification.

What is the best way to quarantine flaky tests in a regression suite?▼

Analyze pass/fail histories to generate quarantine recommendations and apply quarantine annotations by updating tests/regression-suite.md. This stabilizes CI results and maintains a flaky test registry within the regression suite.

When should I run a flaky test scan?▼

Run a flaky test scan during the Polish phase or after multiple CI runs. Analyzing available logs at these stages ensures sufficient pass/fail history exists to accurately detect non-deterministic tests and compute remediation guidance.

How do I get remediation guidance for intermittent test failures?▼

Apply flakiness analysis across multiple CI runs to compute per-test pass/fail trajectories. The scan categorizes detected flaky tests by severity and returns a structured summary including quarantine recommendations and updates to tests/regression-suite.md.

Related Skills

Explore More Software Engineering Skills→
jmaglyjmagly
175

flaky-detect

Analyze CI history and test execution patterns to identify flaky tests.

Community
Advanced
stakworkstakwork
109

ramp

Onboard new legal clinic students through procedures, tools, and practice exercises.

Official
Advanced
XGanQAQXGanQAQ
1

test-flakiness

Analyze CI test logs to detect intermittent PASS/FAIL flaky tests.

Community
Advanced
Euraika-LabsEuraika-Labs
14

test-flakiness

Detect flaky tests in CI logs from GitHub Actions, Unity, Godot, or Unreal.

Official
Advanced