dd-triage-flaky-test

Investigates flaky tests in Datadog CI Visibility and recommends fix, quarantine, or escalation.

1.0k|115|Updated Feb 3, 2026
One-click install
npx skills add https://github.com/datadog-labs/pup --skill dd-triage-flaky-test
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: dd-triage-flaky-test
Source: https://github.com/datadog-labs/pup/tree/main/skills/dd-triage-flaky-test
Command: npx skills add https://github.com/datadog-labs/pup --skill dd-triage-flaky-test

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Flaky tests erode CI trust and waste pipeline time, but triaging them manually requires digging through test history, failure patterns, and blast radius across pipelines. This Skill automates that investigation using Datadog CI Visibility data and produces a structured triage brief with a concrete recommendation.

Core Features & Use Cases

  • Flaky Test Investigation: Retrieves flaky state, failure rate, root-cause category, codeowners, and CI time lost for a specific test via the pup CLI.
  • Failure Pattern & Blast Radius Analysis: Pulls recent failure history, error traces, and counts impacted pipelines to determine whether failures are branch-specific or widespread.
  • Actionable Recommendation: Produces a triage brief recommending fix (with category-specific fix patterns), quarantine (with explicit user approval before writing state changes), or escalation to codeowners.
  • Use Case: A test like TestMyFunc intermittently fails on main. Run this Skill to get its 30-day failure rate, root-cause category (e.g., concurrency), affected pipelines, and a decision on whether to fix the synchronization issue or quarantine the test.

Quick Start

Triage the flaky test TestMyFunc in this repository and tell me whether to fix, quarantine, or escalate it.

Frequently Asked Questions about dd-triage-flaky-test

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I triage a flaky test with Datadog CI Visibility?

Provide the fully qualified test name and repository, and the Skill queries flaky test state, failure rate, and category via pup cicd commands. It then analyzes recent failures and blast radius before recommending fix, quarantine, or escalation.

How to quarantine a flaky test using the pup CLI?

The Skill builds an UpdateFlakyTestsRequest JSON body with the test's fingerprint_fqn and new_state quarantined, then runs pup test-optimization flaky-tests update. It requires explicit user approval before writing and can be reversed by setting new_state back to active.

What information is needed to investigate a flaky test?

You need the fully qualified test name and the repository identifier in lowercase no-schema form like github.com/org/repo. If the repository is not provided, it is derived from git remote get-url origin.

When should a flaky test be quarantined instead of fixed?

Quarantine is recommended when failure rate exceeds 10 percent, blast radius covers more than 5 pipelines, or the root cause category is unknown. Quarantined tests still run but their failures do not block CI.

Why does the flaky test search return no results?

The repository filter may be incorrect. The Skill falls back to searching without a repo filter, extracts git.repository.id_v2 from the results, and retries the query with the confirmed value.

What fixes are not allowed when resolving flaky tests?

Timing hacks like raising timeouts, adding sleeps or retries, and masking changes like relaxing assertions are forbidden. Fixes must eliminate the root cause at the correct layer, whether test, production code, or shared helper.