fixing-flaky-e2e-tests

Diagnose and fix flaky Playwright end-to-end tests in Streamlit CI pipelines.

45.7k|4.4k|Updated Aug 24, 2019
One-click install
npx skills add https://github.com/streamlit/streamlit --skill fixing-flaky-e2e-tests
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: fixing-flaky-e2e-tests
Source: https://github.com/streamlit/streamlit/tree/main/.claude/skills/fixing-flaky-e2e-tests
Command: npx skills add https://github.com/streamlit/streamlit --skill fixing-flaky-e2e-tests

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

End-to-end tests that pass sometimes and fail other times erode trust in CI and waste developer time. This Skill provides a systematic workflow to identify, reproduce, diagnose, and fix flaky Playwright e2e tests in the Streamlit repository.

Core Features & Use Cases

  • Flaky Test Discovery: Runs a script that aggregates playwright_test_stats artifacts from recent CI runs to rank the most flaky tests.
  • Structured Diagnosis Workflow: Guides reproduction with up to 25 local runs, inspection of traces, screenshots, and snapshot diffs, and root-cause identification.
  • Proven Fix Patterns: Documents fixes for timing issues, snapshot mismatches, browser retry behavior, short timeouts, and Firefox subpixel rendering quirks.
  • Use Case: A CI dashboard shows test_date_input failing intermittently on Firefox. Use this Skill to reproduce the failure locally, identify a snapshot timing issue, add an explicit visibility wait, and verify stability with 10 consecutive passing runs.

Quick Start

Ask the assistant to find the top flaky Playwright e2e tests from recent CI runs and fix the most flaky one.

Frequently Asked Questions about fixing-flaky-e2e-tests

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I find the most flaky Playwright tests in CI?

Run the fetch_flaky_tests.py script, which downloads playwright_test_stats artifacts from recent successful CI runs and aggregates tests that required reruns. Use --days, --top, and --min-reruns options to control the lookback window and result count.

How do I fix Playwright snapshot mismatch flakiness?

Compare actual versus expected images in the test-results/snapshot-updates directory to identify timing, animation, or rendering causes. Add explicit waits such as expect(element).to_be_visible() or wait_for_images_loaded() before taking the snapshot.

Why do Playwright tests fail only on Firefox or WebKit?

Firefox has slower console logging, request retries, and subpixel rendering differences, while WebKit has layout timing variations. Fixes include longer timeouts, waiting for image decode, or adding a markdown element to stabilize Firefox subpixel positioning.

When is it acceptable to skip a browser in a flaky test?

Skipping with skip_browser is a last resort, acceptable only after exhausting timing fixes and confirming the flakiness stems from a documented browser limitation rather than a test bug. A clear reason must be included in the marker.

What should I do if a flaky test cannot be reproduced locally?

If the test passes all 25 local reproduction runs, do not attempt a fix. The flakiness likely stems from CI environment factors that cannot be addressed locally, so skip it and move to the next test.