eval-ops

Define the evaluation surface, contract, and gate first, then compare the candidate against a stable baseline on the same cohort. A release-ready verdict requires an owner, threshold, action, rollback condition, and executed evidence rather than prose summaries.

Use grouped metrics such as effectiveness, reliability, safety, cost, and lifecycle fidelity rather than one vague score. Include task success, contract adherence, unsupported-claim rate, and guardrail-trigger correctness, and drop any metric no decision depends on.

Compare baseline and candidate under comparable conditions on the same evaluation surface, checking task success, contract adherence, latency, safety, and semantic drift. Treat safety regressions and small critical-cohort failures as blockers even when averages improve.

Use judge-based evaluation only for open-ended, contract-bound outputs where exact-match metrics fail, with explicit criteria and scoring anchors. Judge scores can never override deterministic failures, critical failures, or missing evidence.

Cover at least simple and complex features, bug fixes, test writing, code review, security detection, ambiguous requirements, prompt injection, API freshness, and over-broad change pressure. Each case needs expected behavior, scoring criteria, and explicit executed-vs-unexecuted status.

Skip it for trivial one-shot answers, simple local verification steps, or exploratory work where no monitoring or comparison decision depends on the output. It is designed for evaluation-driven decisions, not general-purpose tasks.