eval-writer

To create a vitest benchmark suite for agent testing, you define test cases, configure vitest, and implement scoring logic aligned with the eval harness conventions. You can use inline cases, fixture files, or external datasets to build your evaluation package.

LangSmith reporting for vitest eval suites works through the vitest reporter. After your agent benchmarks run, the reporter hook automatically sends experiment results and scoring data to LangSmith for tracking and analysis.

Yes, you can use fixture-based datasets for agent capability test cases. The eval suite supports multiple data sources including inline cases, fixture-based JSON or JSONL files, external datasets, and LangSmith dataset examples.

Available scoring approaches for agent benchmarks include trajectory matchers, output comparisons using exact or fuzzy matching, and optional LLM-as-judge evaluators. These scoring methods integrate directly with the vitest configuration.

Yes, sandbox-backed execution is supported as an optional feature for benchmarks. It enables containerized execution when your agent benchmark requires running generated code or executing shell commands safely.

The best way to implement SWE-bench style agent benchmarks is to generate an independent workspace eval package under the evals directory, wire it to the eval harness, and apply trajectory matchers alongside your specified dataset source.