skills.rest
serpro-workshop-fortaleza
serpro-workshop-fortalezaOFFICIAL
Explore all skills by this author →
GitHub

flaky-test-triage

Diagnose, fix, and quarantine intermittent test failures in CI pipelines.

Updated Sep 10, 2026
Search Tags:#pytest#root cause analysis#flaky tests#playwright#test triage#ci stability#test quarantine
One-click install
npx skills add https://github.com/serpro-workshop-fortaleza/sifap-modernization-paula --skill flaky-test-triage-serpro-workshop-fortaleza
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: flaky-test-triage
Source: https://github.com/serpro-workshop-fortaleza/sifap-modernization-paula/tree/main/.github/skills/flaky-test-triage
Command: npx skills add https://github.com/serpro-workshop-fortaleza/sifap-modernization-paula --skill flaky-test-triage-serpro-workshop-fortaleza

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Intermittent test failures erode trust in CI pipelines and waste developer time on re-runs. This Skill provides a structured workflow to reproduce flaky tests, classify their root cause, fix them properly, or quarantine them under a strict policy. ## Core Features & Use Cases - Reproduction Protocol: Run a suspect test 50+ times in isolation (e.g., pytest --count=50 or Playwright --repeat-each 50) to measure its actual flake rate. - Root-Cause Classification: Categorize instability into async/timing issues, order dependence, external dependencies, non-determinism, or resource contention. - Quarantine Policy: Tests that cannot be fixed within a day get a flaky/ label, a GitHub tracking issue, and a 30-day SLA to fix or delete; quarantined tests run but never fail the build. - Use Case: Your CI fails randomly on an end-to-end test. Use this Skill to reproduce the failure, identify a hardcoded sleep as the cause, replace it with an explicit wait, and log the triage record. ## Quick Start Ask the assistant to triage the flaky test that keeps failing intermittently in CI and decide whether to fix it or quarantine it.

Frequently Asked Questions about flaky-test-triage

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I fix a flaky test that passes on retry?▼

First reproduce it by running the test in isolation 50 times with pytest --count=50 or Playwright --repeat-each 50. Then classify the root cause, such as missing awaits or shared state, and fix that cause rather than adding retries.

What is a test quarantine process for CI?▼

Quarantined tests are labeled with flaky/, tracked in a GitHub issue, and continue running without failing the build. Any test still quarantined after 30 days must be fixed or deleted, since an unfixable test is worse than no test.

Why are sleep statements bad in automated tests?▼

Fixed sleep calls are always incorrect because they either wait too long, slowing suites, or too short, causing intermittent failures. Replace them with explicit waits tied to actual conditions, such as element visibility or network responses.

Should I use retry annotations like @Retry(3) on unstable tests?▼

No. Retry annotations mask instability and reward low-quality tests instead of fixing root causes. The correct approach is reproducing the failure, fixing the underlying timing or isolation issue, or quarantining the test with a deadline.

What flake rate should trigger automatic quarantine?▼

Track each test's failure rate over 100 runs in a dashboard. Automatically quarantine any test exceeding a 5% flake rate, and require a tracking issue plus a 30-day fix-or-delete SLA.

Related Skills

Explore More Software Engineering Skills→
Negentropy-LabyNegentropy-Laby
3

test-flakiness

Analyze CI logs and test histories to identify intermittent failures.

Official
Advanced
Nandansai08Nandansai08
1

e2e-test-triage

Classifies and trims slow, flaky end-to-end test suites into a stabilized PR gate.

Community
Intermediate
manastalukdarmanastalukdar
1

pipeline-monitor

Analyze CI/CD pipeline build success rates, flaky tests, and duration trends.

Community
Advanced
NotysotyNotysoty
9

Flaky Test Analyzer

Analyze test code and failure patterns to categorize flaky test root causes.

Community
Advanced