agent-eval-api

What problem does it solve?

This Skill removes the manual, error-prone work of running Omniloy agent evaluations by guiding you through authentication, resource lookup, test launch, polling, and result inspection.

Core Features & Use Cases

• Reuses existing agents, personas, evaluators, and test configs when they fit, or creates owned resources with the required naming and tagging rules.
• Launches test runs safely with concurrency checks, pre-flight validation, and clear cancellation and polling procedures.
• Supports end-to-end eval workflows such as smoke testing a booking agent, checking tool usage, and reviewing transcripts, scores, and pass or fail outcomes.

Quick Start

Ask the agent-eval-api skill to evaluate the named agent, create or reuse the needed persona and evaluator, launch the run, and report the transcript and scores.