skills.rest
bex-co
bex-coOFFICIAL
Explore all skills by this author →
WebsiteGitHub

routine-flaky-test-fixer

Diagnoses and fixes intermittently failing CI tests by eliminating nondeterminism at its root cause.

275|32|Updated Oct 2, 2020
Search Tags:#debugging#pytest#jest#github actions#flaky tests#ci#test determinism
One-click install
npx skills add https://github.com/bex-co/beancount-io --skill routine-flaky-test-fixer-bex-co
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: routine-flaky-test-fixer
Source: https://github.com/bex-co/beancount-io/tree/main/.agents/skills/routine-flaky-test-fixer
Command: npx skills add https://github.com/bex-co/beancount-io --skill routine-flaky-test-fixer-bex-co

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Flaky tests erode trust in CI pipelines and waste developer time on reruns. This Skill autonomously hunts down intermittently failing tests, identifies the real source of nondeterminism, and fixes it permanently instead of masking it with retries. ## Core Features & Use Cases - CI History Mining: Scans GitHub Actions run history for same-commit red-to-green reruns and inconsistent failure patterns across workflows. - Root-Cause Diagnosis: Identifies common nondeterminism sources such as real timers, missing teardown, shared state, port collisions, unawaited promises, and unseeded randomness. - Determinism Verification: Requires 20 consecutive green runs before shipping, proving the fix rather than relying on luck. - Use Case: A dashboard test fails roughly once in ten CI runs. The Skill mines the CI logs, traces the failure to leaked OTP teardown state, fixes the lifecycle bug, verifies 20 consecutive green runs, and ships the fix via /ship. ## Quick Start Invoke /routine-flaky-test-fixer with an optional package or path to hunt down and fix a flaky test in that scope.

Frequently Asked Questions about routine-flaky-test-fixer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I find and fix flaky tests in CI?▼

Mine GitHub Actions history for same-commit red-to-green reruns using gh run list and gh run view --log-failed, then reproduce locally by rerunning the suspect suite 20-30 times. Fix the root cause of nondeterminism rather than adding retries.

What causes tests to fail intermittently?▼

Common causes include real timers and wall-clock time, missing teardown of handles or temp files, shared state and test-order dependence, port and tmp-path collisions, unawaited promises, network reliance, and unseeded randomness.

Should I use jest.retryTimes to handle flaky tests?▼

No. Retry wrappers and blind timeout bumps mask the flake instead of fixing it, which is worse than leaving it visibly flaky. The correct approach is to fix the nondeterminism itself with fake timers, complete teardown, or awaited async.

How many passing runs prove a flaky test is fixed?▼

Require 20 consecutive green runs of the fixed test, executed in-suite rather than isolated when ordering was implicated. Three passing runs is luck, not determinism.

When is a failing test not considered flaky?▼

A test that fails the same way on every run is a deterministic logic bug, not a flake, and should be handled by a logic-bugfixing routine. Tests that can never fail at all belong to a useless-test pruning routine instead.

Related Skills

Explore More Software Engineering Skills→
Negentropy-LabyNegentropy-Laby
3

test-flakiness

Analyze CI logs and test histories to identify intermittent failures.

Official
Advanced
Obsidian-OwlObsidian-Owl
9

sw-ship

Create a pull request with evidence-mapped body after verifying gates.

Official
Advanced
JuanDiego30JuanDiego30

pr-status-triage

Triage CI failures and PR review comments with blocker-first prioritization.

Community
Intermediate
naxorutonaxoruto

test-flakiness

Analyze CI logs and test histories to detect flaky tests.

Community
Advanced