eval-harness

Automate agent reliability assessments by applying an eval-driven evaluation framework to Claude Code sessions, standardizing capability and regression evals from definition to reporting to establish consistent baselines.

An eval-driven evaluation framework standardizes agent reliability assessments by defining eval plans in a project structure, implementing deterministic code graders, and tracking regression to measure prompt engineering quality.

Set up regression tracking by creating a project structure with .claude/evals definitions, configuring deterministic code graders and model rubrics, and running an initial eval plan to establish a stable baseline for artifact storage.

Yes, you can use model rubrics and human graders together. The framework supports multiple grader types including deterministic code graders, model-based rubrics, and human evaluation across end-to-end workflows.

Yes, a specific project structure is required. You need a .claude/evals directory for eval definitions to enable a stable run and storage pipeline for deterministic code graders and regression tracking.

Establish a baseline for agent capability evaluations by defining eval criteria in the .claude/evals directory, implementing your graders, and running an initial eval plan against your Claude Code setup to capture baseline metrics.