test-diff-analyzer

Flaky tests are identified by comparing test outputs across runs to expose inconsistencies. Capture logs from each run, diff the outputs to find tests that pass sometimes and fail others, then analyze failure patterns to pinpoint which tests fail unpredictably.

Yes. Compare test logs from local runs against CI logs using diff analysis to expose environment-specific failures. The Skill extracts failure indicators and consistency scores to reveal why tests behave differently across local vs CI setups.

A consistency score quantifies how often a test produces the same result across multiple runs. Higher scores indicate stable tests; lower scores flag flaky tests that need investigation and remediation.

Run your test suite and redirect output to log files using commands like `pixi run mojo test > /tmp/test_run_1.log`. Repeat across multiple runs or environments, then use diff to compare outputs and identify instability.

Before/after code-change reconciliation uses diff analysis to separate pre-existing flaky failures from new ones introduced by your changes. Compare baseline test logs against post-change logs to isolate the actual impact of your code.

No. Standard log analysis with grep and sorting (e.g., `grep "FAILED" /tmp/test_run_*.log | sort | uniq -c`) exposes failure patterns. The Skill automates this analysis to compute consistency metrics and provide remediation guidance.