flaky-smoke-tests

Diagnose intermittent VS Code Electron smoke-test failures from Azure DevOps pipeline runs.

190k|41.9k|Updated Sep 3, 2015
One-click install
npx skills add https://github.com/microsoft/vscode --skill flaky-smoke-tests
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: flaky-smoke-tests
Source: https://github.com/microsoft/vscode/tree/main/.github/skills/flaky-smoke-tests
Command: npx skills add https://github.com/microsoft/vscode --skill flaky-smoke-tests

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Intermittent failures in the VS Code Electron smoke-test suite are hard to triage because each platform job runs multiple iterations sequentially, shares a single logs artifact, and uses continueOnError: true, so the overall build result hides which iteration actually failed. This Skill provides a deterministic Azure CLI workflow to isolate the failing iteration, correlate task logs with platform artifacts, and identify the introducing commit.

Core Features & Use Cases

  • Pipeline Inspection: Query Azure DevOps build timelines to find failed Smoke test iteration tasks across macOS, Windows, and Linux jobs.
  • Log Correlation: Download exact iteration task logs and platform-scoped logs-* and crash-dump-* artifacts, then map Mocha failures to smoke-test-runner.log intervals and suite directories.
  • Root Cause Tracing: Walk the gesture → dispatch → request → tool loop → rendering → routing chain, inspect Copilot runtime and Agent Host logs, and use git blame to identify the introducing commit.
  • Validation Runs: Queue focused definition-700 runs with custom iterations parameters on specific platforms to confirm fixes.

Quick Start

Diagnose the latest failed Flaky Smoke Tests build for VS Code by inspecting its timeline, downloading the failing iteration's task log and platform logs artifact, and identifying the introducing commit.

Frequently Asked Questions about flaky-smoke-tests

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I find which smoke-test iteration failed in Azure DevOps?

Use `az devops invoke` with the build timeline resource and filter records whose names contain `Smoke test iteration`. Treat `succeededWithIssues`, `failed`, and tasks with error issues as failed iterations, since `continueOnError: true` masks failures in the overall build result.

How to download VS Code smoke-test logs from Azure DevOps using CLI?

Use `az pipelines runs artifact download` for platform-scoped `logs-*` artifacts and `az devops invoke` with the `logs` resource (not `buildLog`) for individual task logs. Convert the JSON `value` array to plain text using `jq` or PowerShell `ConvertFrom-Json`.

Why does the smoke-test build show succeeded but tests actually failed?

The Flaky Smoke Tests pipeline uses `continueOnError: true` on iteration tasks, so failed iterations report as `succeededWithIssues` and later iterations still run. You must inspect every iteration task in the timeline rather than the overall build or job result.

Where are Copilot runtime logs stored in VS Code smoke test artifacts?

Copilot runtime process logs are captured at `copilot-runtime-logs/process-*.log` inside each suite directory by `dumpFailureDiagnostics`. Agent Host sessions write full trace logs there, while Chat Sessions and Local sessions write only minimal startup logs.

Can I queue a custom smoke-test run with specific iterations?

Yes, use `az pipelines run --id 700` with the `iterations` parameter as a JSON array like `[1,2,3,4,5,6]`. Enable only relevant platforms via `VSCODE_BUILD_MACOS`, `VSCODE_BUILD_LINUX`, and `VSCODE_BUILD_WIN32` parameters, and cancel obsolete runs on the same branch first.

What is the difference between task log and platform logs artifact?

The task log is iteration-scoped and authoritative for the failure summary and time range. The platform logs artifact is job-scoped and shared across iterations, so later iterations can overwrite files. Use a one-iteration diagnostic run when detailed per-iteration diagnostics are required.