skills.rest
EntityProcess
EntityProcessOFFICIAL
Explore all skills by this author →
WebsiteGitHub

agentv-eval-orchestrator

Orchestrate AgentV evaluations by simulating LLM responses without API keys.

15|3|Updated Nov 7, 2025
Search Tags:#cli#llm#evaluation#orchestration#prompt testing#agentv
One-click install
npx skills add https://github.com/EntityProcess/agentv --skill agentv-eval-orchestrator
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: agentv-eval-orchestrator
Source: https://github.com/EntityProcess/agentv/tree/main/plugins/agentv-dev/skills/agentv-eval-orchestrator
Command: npx skills add https://github.com/EntityProcess/agentv --skill agentv-eval-orchestrator

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill enables users to run AgentV evaluations by acting as the LLM themselves, eliminating the need for API keys and external service configurations.

Core Features & Use Cases

  • API-Key Free Evaluation: Run AgentV evaluations locally by simulating the LLM's response.
  • Step-by-Step Orchestration: Guides the user through the process of getting task input, executing the task as the LLM, and judging the result.
  • Use Case: You need to test a new prompt for your agent but don't have API keys set up for AgentV. Use this skill to manually perform the evaluation steps.

Quick Start

Use the agentv-eval-orchestrator skill to run an eval file named 'my_eval.yaml'.

Frequently Asked Questions about agentv-eval-orchestrator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I test agent prompts locally without an LLM API key?▼

You can test agent prompts locally without an API key by manually simulating LLM responses. This orchestrator guides you through input, task execution, and judging steps to evaluate prompt quality interactively.

What is manual prompt evaluation in CLI workflows?▼

Manual prompt evaluation in CLI workflows is a process where you act as the LLM to generate task responses and judge outcomes. It enables local prompt testing without external service configurations or API authentication.

How do I run AgentV evaluations step by step?▼

To run AgentV evaluations, execute the prompt eval subcommands sequentially for task input, task execution, and result judging. The orchestrator directs this interactive workflow to complete the local evaluation cycle.

Can I use my own eval files for prompt testing with this orchestrator?▼

Yes, you can use your own eval files for prompt testing. Simply specify your eval file, such as 'my_eval.yaml', when initiating the orchestration workflow to test your custom prompts.

Do I need external service configurations to judge prompt quality?▼

No, you do not need external service configurations to judge prompt quality. The evaluation mechanism relies on manual execution and human judgment to assess results locally without connecting to external LLM APIs.

Related Skills

Explore More Software Engineering Skills→
spulido99spulido99
2

Evals-Driven Development

Define scenarios, execute tests, and analyze results for agent evaluations.

Community
Advanced
softee-psoftee-p

skill-creator

Compile user ideas into reproducible Claude Skill units with SKILL.md and evaluations.

Community
Advanced
baekenoughbaekenough
13

rtk-exec

Proxy CLI commands through RTK to compress stdout and stderr.

Community
Intermediate
coval-aicoval-ai
2

review-llm-annotations-and-improve-prompt

Analyze human versus LLM judge annotation discrepancies to refine evaluation prompts.

Official
Advanced