skills.rest
microsoft
microsoftOFFICIAL
Explore all skills by this author →
Website𝕏@OpenAtMicrosoftGitHub

pipeline-test-triage

Triage Azure DevOps pipeline test failures and classify flaky versus deterministic bugs.

4.9k|585|Updated Aug 22, 2019
Search Tags:#pipeline#Azure DevOps#mocha#fluid framework#test triage#CI failure analysis#Flaky tests
One-click install
npx skills add https://github.com/microsoft/FluidFramework --skill pipeline-test-triage
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: pipeline-test-triage
Source: https://github.com/microsoft/FluidFramework/tree/main/.claude/skills/pipeline-test-triage
Command: npx skills add https://github.com/microsoft/FluidFramework --skill pipeline-test-triage

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

Investigating a failed Azure DevOps pipeline run means digging through hundreds of logs and thousands of lines, most of which are passing tests. This Skill analyzes Fluid Framework ADO pipeline runs, classifies each failure as service flakiness, infrastructure issues, or a deterministic product bug, and distinguishes flaky tests from real regressions.

Core Features & Use Cases

  • Tiered failure analysis: Starts with the cheapest source (Playwright on the Tests tab), falls back to ADO MCP log parsing, then targeted Test Results/Timeline REST endpoints, avoiding bulk log downloads.
  • Flaky vs. deterministic verdicts: Compares the same test across adjacent runs and per-test history to decide whether a failure is a real bug worth filing.
  • Hidden failure detection: Surfaces failures buried in retried attempts of otherwise green builds using timeline sub-timelines.
  • Bug filing: Produces ready-to-paste ADO bug descriptions with direct error-line deep links.
  • Use Case: A developer sees the Real Service End to End Tests pipeline failed and asks why; the Skill identifies the failing driver leg, parses the mocha report log, compares prior runs, and reports that the failure is a deterministic odsp bug with a drafted work item.

Quick Start

Analyze the test failures in ADO build 409255 of the Real Service End to End Tests pipeline and tell me which failures are flaky and which need bugs filed.

Frequently Asked Questions about pipeline-test-triage

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I find out why an Azure DevOps pipeline failed?▼

Start with the Tests tab in the ADO build results page, filtered to failed outcomes, which groups failures with error messages and stacks. If no browser session is available, use the ADO MCP build status tool to list failing legs, then parse the relevant report log for the mocha failure block.

How to tell if a failing test is flaky or a real bug?▼

Compare the same test across adjacent pipeline runs: a deterministic bug fails with the same assertion and stack in every run where the task executed, while flaky tests appear intermittently with varying timeout or service errors. Per-test history endpoints or the Tests tab sparkline answer this directly.

Can a green Azure DevOps build still hide test failures?▼

Yes. A test that fails on Attempt 1 and passes on auto-retry leaves the build green with zero reported failures. The failure is found by querying timeline sub-timelines for records with previousAttempts and filtering for failed tasks.

What authentication is needed for Azure DevOps REST API calls?▼

Use an AAD token from Azure CLI via az account get-access-token with the ADO resource ID, or a least-privilege PAT with Build Read and Test Read scopes stored in an environment variable. The ADO MCP tools are pre-authenticated and need no setup.

Why does bulk-downloading pipeline logs not work well for triage?▼

A single run produces around 660 logs totaling tens of megabytes, mostly passing container output, and the biggest logs are usually passing legs rather than failing ones. Targeted report-log fetching or Test Results REST endpoints return only the relevant few kilobytes.

Related Skills

Explore More Software Engineering Skills→
DataDogDataDog
570

analyze-azdo-build

Analyze Azure DevOps CI build failures and categorize issues by pull request or build ID.

Official
Advanced
microsoftmicrosoft
4.9k

trigger-pipelines-for-copilot-pr

Trigger Azure DevOps pipelines for Copilot pull requests via GitHub comments.

Official
Basic
uzumaki-incuzumaki-inc

health

Run code health checks and compute a weighted 0-10 score.

Official
Advanced
reliant-labsreliant-labs
4

reproduce

Reproduce software bugs with diagnostic logging and minimal e2e triggers.

Official
Intermediate