investigate-golden-flake

Diagnose failing golden tests by analyzing CI logs and diffs.

64|8|Updated Oct 24, 2025

Search Tags:#debugging #logs #diff #ci #replay #golden-test

Ready to run

npx skills add https://github.com/GregorStocks/mage-bench --skill investigate-golden-flake

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: investigate-golden-flake
Source: https://github.com/GregorStocks/mage-bench/tree/main/.claude/skills/investigate-golden-flake
Command: npx skills add https://github.com/GregorStocks/mage-bench --skill investigate-golden-flake

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This skill helps engineers quickly diagnose failing golden tests by extracting failure details from CI logs, diffs, and RPC traces, and guides them to the root cause.

Core Features & Use Cases

Analyze GitHub Actions golden-test failures by parsing run URLs, logs, and difference outputs.
Classify failures into common categories such as deterministic prompt drift, race conditions, or nondeterministic ordering.
Provide actionable steps and best-practice fixes for bridge/LMS test harness flakiness and replay reproducibility.

Quick Start

Provide the GitHub Actions URL for the failing golden test run and I will fetch the logs, identify the failing diff, and guide you to the root cause.

Frequently Asked Questions about investigate-golden-flake

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I diagnose failing golden tests in GitHub Actions CI pipelines?▼

Diagnose failing golden tests by providing the GitHub Actions run URL to fetch CI logs, analyze diffs and RPC traces, classify failures like prompt drift or race conditions, and receive actionable guidance to fix root causes.

What causes flaky golden test failures in CI?▼

Flaky golden test failures are caused by deterministic prompt drift, race conditions, and nondeterministic ordering. Analyzing CI logs, diffs, and test replays helps classify these common bridge and LMS test harness flakiness patterns.

How do I fix nondeterministic ordering and race conditions in golden tests?▼

Fix nondeterministic ordering and race conditions by using testing utilities for normalization and replay. Analyzing RPC traces and diffs from failing CI runs guides the application of best-practice fixes for test harness flakiness and reproducibility.

Can I use CI logs and diffs to identify prompt drift in golden tests?▼

Yes, CI logs and diffs can identify deterministic prompt drift in golden tests. Parsing the failing run URL extracts difference outputs and RPC traces, classifying the failure and guiding fixes for bridge and LMS test harness reproducibility.

Do I need access to failing run URLs to debug golden test flakes?▼

Yes, access to failing GitHub Actions run URLs is required to fetch logs and diffs. Familiarity with the bridge, golden test flake patterns, and testing utilities used for normalization and replay is also needed to guide fixes.

What's the best way to troubleshoot golden test flakiness without repro locally?▼

Troubleshoot golden test flakiness by analyzing CI logs, diffs, and RPC traces from the failing run URL. This approach classifies failures into categories like race conditions and provides best-practice fixes using replay and normalization utilities.