cross-run-consistency

Execute tests N times and diff outputs to classify non-determinism.

1|Updated Apr 12, 2026

Search Tags:#testing #determinism #ci #regression-testing #nondeterminism #cross-run

Ready to run

npx skills add https://github.com/mytechsonamy/VibeFlow --skill cross-run-consistency

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: cross-run-consistency
Source: https://github.com/mytechsonamy/VibeFlow/tree/main/skills/cross-run-consistency
Command: npx skills add https://github.com/mytechsonamy/VibeFlow --skill cross-run-consistency

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Runs the same test N times in one session, diffs the outputs, and classifies non-determinism by root cause. Complements observability's historical flake tracking with an immediate "does this test agree with itself right now?" answer. Gate contract — P0 scenarios must be strict-consistent (same output on N/N runs), no tolerance fuzzing, no silent averaging. PIPELINE-5 step 3.

Core Features & Use Cases

Cross-run reproducibility: run the critical tests multiple times and compare outputs in a single session.
Determinism classification: categorize differences by root cause using a predefined taxonomy.
On-demand gate checks: perform cross-run checks before releases and during regression-workflows.

Quick Start

Execute the cross-run consistency command with a scenario glob and optional --runs and --mode flags to begin the session.

Frequently Asked Questions about cross-run-consistency

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I detect non-determinism in my regression tests?▼

Detect test non-determinism by executing the same tests N times in a single session and diffing the outputs. This approach classifies differences by root cause taxonomy, providing immediate cross-run reproducibility validation for regression suites.

What is cross-run consistency checking in CI pipelines?▼

Cross-run consistency checking executes critical tests multiple times within a single session to compare and diff outputs. It enforces strict mode for P0 scenarios, ensuring identical results across N/N runs without tolerance fuzzing or silent averaging before releases.

How do I classify flaky test root causes by diffing outputs?▼

Classify flaky test root causes by capturing a baseline output, executing sequential runs with artifact collection, and applying diff logic against a predefined taxonomy. The process outputs a consistency report detailing the classified non-determinism.

Can I run on-demand determinism checks across different code changes?▼

Yes, you can run on-demand determinism checks across code changes and environments. By applying per-test mode resolution (strict or tolerant), the checks validate cross-run reproducibility and output a consistency report with artifacts for regression workflows.

What is the difference between strict and tolerant test consistency modes?▼

Strict mode requires identical outputs across all N runs, typically gating P0 scenarios with zero tolerance fuzzing. Tolerant mode allows per-test resolution for non-critical tests, capturing artifacts and classifying differences without failing the entire consistency check.

When should I use an immediate cross-run check instead of historical flake tracking?▼

Use an immediate cross-run check when you need to verify if a test agrees with itself right now, complementing historical flake tracking. It is ideal for on-demand gate checks before releases and during regression workflows to ensure strict consistency.