dd-triage-flaky-test

Investigates flaky tests via Datadog CI Visibility and recommends fix, quarantine, or escalation.

5|2|Updated Mar 9, 2026
One-click install
npx skills add https://github.com/lauhon/pi --skill dd-triage-flaky-test-lauhon
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: dd-triage-flaky-test
Source: https://github.com/lauhon/pi/tree/main/skills/dd-triage-flaky-test
Command: npx skills add https://github.com/lauhon/pi --skill dd-triage-flaky-test-lauhon

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Flaky tests erode CI trust and waste pipeline time, but triaging them manually requires digging through history, failure patterns, and blast radius across many pipelines. This Skill automates that investigation using Datadog CI Visibility data and produces a structured triage brief with a concrete recommendation. ## Core Features & Use Cases - Flaky Test Investigation: Retrieves flaky state, failure rate, category, codeowners, and lost CI time for a specific test via the pup CLI. - Failure Pattern Analysis: Pulls recent failure history, error traces, and branch distribution to identify root causes and compute blast radius across impacted pipelines. - Actionable Recommendations: Applies category-specific fix patterns (concurrency, timeout, network, etc.) with strict guardrails against timing hacks, and can quarantine a test through the Datadog API after explicit user approval. - Use Case: A test like TestMyFunc intermittently fails in CI. Run this Skill to get its failure rate, root-cause category, affected pipelines, and a decision on whether to fix the code, quarantine the test, or escalate to codeowners. ## Quick Start Triage the flaky test TestMyFunc in this repository and tell me whether to fix, quarantine, or escalate it.

Frequently Asked Questions about dd-triage-flaky-test

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I triage a flaky test with Datadog CI Visibility?

Provide the fully qualified test name and repository, and the Skill queries flaky test state, failure rate, category, and recent failure history via the pup CLI. It then produces a triage brief recommending fix, quarantine, or escalation.

How to quarantine a flaky test using the Datadog API?

The Skill builds an UpdateFlakyTestsRequest with the test's fingerprint_fqn and new_state quarantined, then runs pup test-optimization flaky-tests update. Quarantine only proceeds after explicit user approval, and setting new_state back to active reverses it.

What information do I need to investigate a flaky test?

You need the fully qualified test name and the repository in lowercase no-schema URL form, which can be derived from git remote get-url origin. The pup CLI must be installed and authenticated to Datadog.

When should a flaky test be quarantined instead of fixed?

Quarantine is recommended when failure rate exceeds 10 percent, blast radius covers more than 5 pipelines, or the root cause cannot be confirmed from the stack trace. Tests with unknown category and low failure rate are escalated to codeowners instead.

Why does the flaky test search return no results?

The repository filter may be incorrect. The Skill falls back to searching without a repo filter, extracts git.repository.id_v2 from the results, and retries the query with the confirmed repository identifier.

What fixes are not allowed when addressing flaky tests?

Timing hacks such as raising timeouts, adding sleeps, or retries are forbidden, as are masking changes like relaxing assertions. Fixes must eliminate the root cause at the correct layer and update all call sites of shared helpers.