skills.rest
kortix-ai
kortix-aiOFFICIAL
Explore all skills by this author →
Website𝕏@kortixGitHub

flaky-test-quarantine

Detects flaky tests from CI run history and quarantines them via skip-marker pull requests.

20.2k|3.4k|Updated Oct 5, 2024
Search Tags:#triage#Automation#ci#GitHub actions#test quarantine#Flaky tests#Pull request
One-click install
npx skills add https://github.com/kortix-ai/suna --skill flaky-test-quarantine
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: flaky-test-quarantine
Source: https://github.com/kortix-ai/suna/tree/main/packages/starter/templates/marketplace/runtime/skills/flaky-test-quarantine
Command: npx skills add https://github.com/kortix-ai/suna --skill flaky-test-quarantine

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Flaky tests that pass and fail non-deterministically erode trust in CI and block merges, but manually tracking them across runs is tedious and error-prone. This Skill automates daily triage: it scores each test's non-determinism against a persistent ledger and quarantines the worst offenders without ever deleting them.

Core Features & Use Cases

  • Evidence-based flakiness scoring: Pulls CI run history from GitHub, parses JUnit/XML test reports, and computes flip counts and failure rates over a rolling window stored in a durable per-test ledger.
  • Reversible quarantine PRs: Adds framework-appropriate skip markers (Jest/Vitest, Pytest, Go) with reasons and issue links, opens one PR per run, and never deletes test code or merges on its own authority.
  • Tracking and reporting: Maintains a single running GitHub tracking issue with a quarantine table and posts a concise summary to Slack, including candidates stable enough for de-quarantine.
  • Use Case: A platform team schedules this Skill daily against their monorepo; after two weeks it identifies three tests that flip outcomes on unchanged code, opens a quarantine PR skipping them with evidence links, and posts the summary to the team's Slack channel.

Quick Start

Ask the agent to run the flaky test triage sweep against your repository and quarantine any tests that have crossed the flakiness threshold.

Frequently Asked Questions about flaky-test-quarantine

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automatically quarantine flaky tests in CI?▼

Run this Skill on a daily schedule against your repository. It pulls CI run history via the GitHub CLI, scores each test's flakiness from a persistent ledger, and opens a pull request that adds skip markers with reasons to tests crossing the threshold.

How is a flaky test score calculated from CI runs?▼

The score weights flip count above raw failure rate: flips are outcome changes between consecutive runs of the same or near-identical code. A test failing consistently on broken code is not flaky, and a single failure never triggers quarantine.

Does flaky test quarantine work with Pytest, Jest, and Go?▼

Yes, the Skill adapts its skip marker to the project's test framework: it.skip for Jest/Vitest, @pytest.mark.skip with a reason for Pytest, and t.Skip for Go. It only adds markers and never removes test bodies or assertions.

Can a quarantined test be restored later?▼

Yes, quarantine is reversible by construction since only a skip marker is added. The tracking issue lists stable candidates whose recent ledger runs are clean, and deleting the skip marker restores the test after a human review.

Will the agent merge quarantine PRs or modify CI configuration?▼

No, the agent opens one pull request per run and stops; a human reviews and merges. It never pushes to main, edits CI workflow files, disables suites, or retries until green.

Related Skills

Explore More Software Engineering Skills→
nevesbnevesb

test-flakiness

Analyze JUnit XML and CI logs to classify flaky tests.

Community
Intermediate
sast-playgroundsast-playground

test-management

Automate quarantining and re-enabling tests in the Aspire repository with QuarantineTools.

Official
Intermediate
manastalukdarmanastalukdar
1

pipeline-monitor

Analyze CI/CD pipeline build success rates, flaky tests, and duration trends.

Community
Advanced
aquariuscookaquariuscook

QE Test Execution

Orchestrate parallel test execution with retry logic and result aggregation.

Community
Advanced