agent-eval-harness

To evaluate AI agent interactions, this Skill runs prompts against headless adapters and captures full interaction trajectories, including thoughts, tools, and plans, into standardized JSONL outputs for downstream scoring.

You can compare AI agent configurations by using the built-in compare command, which summarizes and evaluates standardized JSONL outputs across multiple runs to highlight performance differences.

Yes, this agent trajectory evaluation requires a Bun-compatible runtime, along with project schemas, headless adapters, and a configured execution environment to run the capture, extract, and grade tasks.

Calibrating graders for AI agent outputs is handled by a dedicated CLI command that applies schema-driven adapters to standardize interaction data, ensuring accurate and consistent scoring across evaluation runs.

Yes, you can integrate AI agent evaluation pipelines into CI. This framework is pipeline-friendly, supporting CLI commands like capture, grade, and summarize to automate standardized JSONL output generation within continuous integration workflows.

Validating reference solutions in AI agent evaluation ensures your grading mechanisms reliably assess agent performance against known correct outcomes. This Skill supports reference-solution validation within schema-driven environments to verify evaluation accuracy.