agent-eval-api

Name: agent-eval-api
Availability: InStock
Author: Omniloy

Orchestrate Omniloy agent evaluation runs from resource discovery to result retrieval.

Updated Jun 22, 2026

Search Tags:#personas #scoring #transcripts #evaluators #api-automation #test-runs #agent-evals

Ready to run

npx skills add https://github.com/Omniloy/agent-skills --skill agent-eval-api

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: agent-eval-api
Source: https://github.com/Omniloy/agent-skills/tree/main/skills/agent-eval-api
Command: npx skills add https://github.com/Omniloy/agent-skills --skill agent-eval-api

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill removes the manual, error-prone work of running Omniloy agent evaluations by guiding you through authentication, resource lookup, test launch, polling, and result inspection.

Core Features & Use Cases

Reuses existing agents, personas, evaluators, and test configs when they fit, or creates owned resources with the required naming and tagging rules.
Launches test runs safely with concurrency checks, pre-flight validation, and clear cancellation and polling procedures.
Supports end-to-end eval workflows such as smoke testing a booking agent, checking tool usage, and reviewing transcripts, scores, and pass or fail outcomes.

Quick Start

Ask the agent-eval-api skill to evaluate the named agent, create or reuse the needed persona and evaluator, launch the run, and report the transcript and scores.

Frequently Asked Questions about agent-eval-api

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I run end-to-end agent evaluations using an API?▼

You can run end-to-end agent evaluations by authenticating with the testing platform API, discovering existing resources, launching test runs with pre-flight validation, and polling until terminal states to retrieve scores and transcripts.

What is the process for reusing existing personas and evaluators in test runs?▼

Reusing existing personas and evaluators in test runs requires a strict GET-before-create resolution pattern to discover available resources before deciding whether to reuse them or create new AI-generated prefixed owned resources.

How do I safely launch agent test runs with concurrency and timeout checks?▼

Safely launching agent test runs requires pre-flight validation, concurrency checks, and clear cancellation procedures to ensure the testing platform handles multiple test runs without exceeding operational limits.

How do I retrieve pass or fail outcomes and transcript evidence from agent evaluations?▼

Retrieving pass or fail outcomes and transcript evidence involves polling test runs until they reach a terminal state, then reporting the final score, pass or fail status, and the associated transcript data.

Do I need to create new test configurations for every agent evaluation workflow?▼

You do not need to create new test configurations for every agent evaluation workflow; the system reuses existing test configs when they fit, applying required naming and tagging rules only when creating owned resources.