octocode-graph-eval

Freeze the eval harness, record a baseline under a fixed budget, make the smallest subject change, then re-measure with the same command. Keep the change only if the primary KPI improves and guardrails hold; otherwise discard it and log the hypothesis.

Set the primary KPI at the graph boundary, add per-node sensors, and run edge detection to confirm real dependencies before parallelizing. Attribute failures by bisecting nodes with frozen inputs, and require fresh-context verifiers plus at least one anchor node with deterministic evidence.

No. Public benchmarks are orientation signals only because of contamination and saturation risk. Build a private eval suite from your own real failure traces and use it as the ship gate, treating public scores as weak evidence without transcript audits.

Goodhart's Law means a loop optimizing one metric can hit it while the real goal degrades, such as resolution rate rising while satisfaction falls. Assign every primary KPI a counter-metric guardrail the agent cannot tune, and reframe the goal when the guardrail degrades.

Escalate when the primary metric stays flat across many trials and error analysis finds no new failure categories. A bilevel outer loop reads the inner loop's execution trace and rewrites the search strategy itself, rather than running more inner trials or editing graders.

A verifier receiving the executor's full conversation history is agreeing with itself in a different font, not verifying. Require each verifier node to start with fresh context and grade only the artifact before calling a result verified.