magpie-flaky-test-triage

To detect flaky tests in GitHub Actions, analyze CI run outcomes to identify intermittent failures and compute failure rates for individual jobs. This classifies jobs as flaky, consistently broken, or clean without modifying test code or workflow files.

Flaky jobs exhibit intermittent test failures across CI runs, while consistently broken jobs fail every time. Analyzing GitHub Actions CI run history computes these failure rates to distinguish intermittent test failures from permanent breakages.

Audit repository health for intermittent test failures by running the triage script with the repository name and a time window, such as 30 days. It analyzes GitHub Actions CI run history to output a prioritized list of flaky jobs for review.

Yes, identifying intermittent test failures requires access to the GitHub API and the ability to parse CI run data. This access is necessary to analyze GitHub Actions run history and classify jobs based on their failure rates.

Yes, you can triage flaky tests without modifying test code or workflow files. The analysis parses GitHub Actions CI run data to provide a prioritized list of flaky jobs for review, leaving your repository workflows untouched.

The best way to prioritize intermittent test failures is to compute failure rates for individual jobs across GitHub Actions CI runs. This triage output distinguishes flaky jobs from consistently broken ones, providing a prioritized list for maintainers to review.