skills.rest
youdotcom-oss
youdotcom-ossOFFICIAL
Explore all skills by this author →
Website𝕏@youdotcomGitHub

agent-eval-harness

Capture full conversational agent trajectories into structured JSONL files.

5|Updated Jan 20, 2026
Search Tags:#evaluation#agents#benchmark#bun#jsonl#grading#headless-adapters
One-click install
npx skills add https://github.com/youdotcom-oss/web-search-agent-evals --skill agent-eval-harness-youdotcom-oss
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: agent-eval-harness
Source: https://github.com/youdotcom-oss/web-search-agent-evals/tree/main/.plaited/skills/agent-eval-harness%40plaited_agent-eval-harness
Command: npx skills add https://github.com/youdotcom-oss/web-search-agent-evals --skill agent-eval-harness-youdotcom-oss

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

Captures full agent execution traces from headless CLI agents to enable reproducible evaluation and scoring workflows.

Core Features & Use Cases

  • Capture complete trajectories (thoughts, tool calls, plans, messages) for downstream grading and benchmarking.
  • Schema-driven adapters to run different headless agents via Bun, Claude, Gemini, Droid, and You.com MCP integrations.
  • Produce a single JSONL trajectory file that can feed graders, calibration, and comparison analyses.

Quick Start

Run a sample capture with a headless adapter to generate results.jsonl.

Frequently Asked Questions about agent-eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I capture agent trajectories for benchmarking and evaluation?▼

You can capture agent trajectories by executing prompts against headless adapters and recording the full execution traces into a structured JSONL file for downstream grading. This skill automates that end-to-end capture process.

What is the best way to evaluate conversational agents using headless CLI adapters?▼

Evaluating conversational agents is best done via schema-driven adapters that run headless CLIs, capturing complete thoughts, tool calls, and messages into JSONL for comparative analysis and scoring.

Do I need Bun and Node.js to run agent evaluation captures?▼

Yes, you need both Bun and Node.js environments to orchestrate the captures, execute prompts against compatible headless adapters, and generate the structured trajectory outputs.

Can I use MCP tool configurations when benchmarking multiple agents?▼

Yes, you can apply MCP tool configurations when benchmarking multiple agents. The harness supports schema-driven adapters for various integrations, capturing complete trajectories for comparative analysis.

What file format is produced for downstream agent scoring and calibration?▼

The process produces a single JSONL trajectory file containing complete execution traces, which can then feed directly into graders, calibration workflows, and comparison analyses.

Related Skills

Explore More Software Engineering Skills→
plaitedplaited
6

agent-eval-harness

Capture AI agent trajectories and emit standardized JSONL outputs.

Official
Advanced
thecloudtipsthecloudtips
18

Data Analysis

Analyze structured datasets for statistics, trends, outliers, and correlations.

Community
Advanced
youdotcom-ossyoudotcom-oss
59

ydc-claude-agent-sdk-integration

Integrate Claude Agent SDK with You.com's HTTP MCP server for Python and TypeScript.

Official
Intermediate
NicolasPadovamNicolasPadovam

opensquad-skill-creator

Develop, evaluate, and optimize custom agent skills within the Opensquad framework.

Community
Advanced