agent-eval-tests

What problem does it solve?

Writing reliable evals for AI agents is hard: trusting the agent's self-reported success is unreliable, deterministic checks are tricky to wire against a live database, and LLM-judge scorers are easy to misuse. This Skill provides the patterns, APIs, and hard-won debugging knowledge for building evalite-scored agent evals in the DXOS assistant-evals package.

Core Features & Use Cases

• Eval authoring with createEvalRunner: Boot a full Composer test harness, run a real prompt against the live agent stack, and grade outcomes with scorers.
• Deterministic DB assertions: Use objectExists, findObject, completedBlocks, and toolInvocations to verify real database effects and tool calls instead of trusting agent self-reports.
• LLM-judge scorers: Apply the judge() helper for open-ended quality criteria, with guidance on demonstrating failure cases and avoiding overuse.
• Use Case: You need to add an eval verifying the agent creates an Organization object in the database. Use this Skill to scaffold the eval file, add a dbQuery assertion, configure timeouts, and avoid known pitfalls like stale dist imports or tool-name mismatches.

Quick Start

Ask the AI to create a new eval file in packages/core/compute/assistant-evals/src/evals that verifies the agent creates a named object in the database using createEvalRunner and an objectExists assertion.