eval-engineering

Follow the eval-engineering flow: inspect the repository and traces, draft a reviewed Task.md spec, then build the Harbor package with task.toml, instruction.md, an environment definition, and tests/test.sh as the verifier entry point. Audit the package before any scored model run.

Build independent checks that read final environment state rather than agent claims, and test the verifier with a valid result, a valid alternative, a wrong result, a shortcut, a collateral change, and missing evidence. Every completed verifier path must write a parseable reward without exposing hidden truth.

It is a repository-local skill that stores reusable project-specific knowledge such as environment methods, data creation patterns, and verifier evidence. It helps generate future Task Specs while exact task truth stays in the collocated Task.md file.

Yes, it includes a multi-turn simulation reference with a runner, model user, and Harbor adapter example. It requires proving the first input matches instruction.md, later turns follow the declared user policy, and one approved session is reused.

Pass rates alone cannot prove task quality because weak models can pass through leakage and strong models can fail from broken environments. Calibration requires reading complete trajectories and classifying each failure as capability, harness, environment, verifier, leakage, or infrastructure.

Model comparison is optional and only useful when it answers a named uncertainty, such as an unclear instruction or a weak verifier. One carefully inspected target-model run can be sufficient when deterministic checks pass.