counterfactual-failure-analysis

Counterfactual failure analysis evaluates why a coding-agent run failed by testing alternative evidence or context interventions. It compares replay outputs against observed failures to determine what changed and what should be tested or falsified.

To write an evidence memo, you collect ee lab capture, replay, and counterfactual JSON outputs for the same workspace and episode. You then separate observed facts, replay-supported claims, hypotheses, and assumptions into a structured document.

Yes, you can use strictly parsed ee lab JSON outputs containing status, capture, replay, counterfactual, and why fields. The analysis requires provenance linkage and redaction verification to ensure evidence is bound to the specific workspace.

Replay validation is required to enforce Stop/Go guardrails against overclaiming. The analysis refuses to conclude success without replay-supported claims, preventing speculative conclusions about why an agent run failed.

The analysis enforces strict boundaries preventing durable mutation or DB scraping and requires degraded-code preservation. It cannot conclude success without replay support, and all evidence must pass redaction compliance and workspace scoping checks.

The best way to falsify causal hypotheses is to ground them in explicit ee lab evidence rather than speculation. You produce an evidence-linked memo with explicit falsification paths separating verified replay claims from unverified assumptions.