Orchestrate Omniloy agent evaluation runs from resource discovery to result retrieval.
npx skills add https://github.com/Omniloy/agent-skills --skill agent-eval-api
Please help me install this Agent Skill. Skill: agent-eval-api Source: https://github.com/Omniloy/agent-skills/tree/main/skills/agent-eval-api Command: npx skills add https://github.com/Omniloy/agent-skills --skill agent-eval-api
This Skill removes the manual, error-prone work of running Omniloy agent evaluations by guiding you through authentication, resource lookup, test launch, polling, and result inspection.
Ask the agent-eval-api skill to evaluate the named agent, create or reuse the needed persona and evaluator, launch the run, and report the transcript and scores.
High-intent search queries and answers about installing and using this skill.
You can run end-to-end agent evaluations by authenticating with the testing platform API, discovering existing resources, launching test runs with pre-flight validation, and polling until terminal states to retrieve scores and transcripts.
Reusing existing personas and evaluators in test runs requires a strict GET-before-create resolution pattern to discover available resources before deciding whether to reuse them or create new AI-generated prefixed owned resources.
Safely launching agent test runs requires pre-flight validation, concurrency checks, and clear cancellation procedures to ensure the testing platform handles multiple test runs without exceeding operational limits.
Retrieving pass or fail outcomes and transcript evidence involves polling test runs until they reach a terminal state, then reporting the final score, pass or fail status, and the associated transcript data.
You do not need to create new test configurations for every agent evaluation workflow; the system reuses existing test configs when they fit, applying required naming and tagging rules only when creating owned resources.
langchain-aiAutomate agent output evaluation with LangSmith evaluators in Python and TypeScript.
yoshisadaValidate Claude Code skills using the Kiln task-shapes testing framework.
h4vzzAutomate repetitive tasks using scripts, file watchers, scheduling, and API polling.
lossyrobValidate PAW PRs and generate scalable descriptions with merge guidance.
