Develops and executes agent behavior evaluations using a YAML-based test case definition system.
npx skills add https://github.com/orient-bot/orient --skill eval-testing
Please help me install this Agent Skill. Skill: eval-testing Source: https://github.com/orient-bot/orient/tree/main/.claude/skills/eval-testing Command: npx skills add https://github.com/orient-bot/orient --skill eval-testing
This Skill provides a robust framework for developing, running, and analyzing evaluations of agent behavior, ensuring consistency and quality across different prompts, tools, models, and configurations.
Run all agent behavior evaluations using the default model.
High-intent search queries and answers about installing and using this skill.
You can evaluate agent behavior by creating YAML-based test case definitions that specify inputs, expected outcomes, and mock services. These definitions allow you to systematically verify tool selection, response quality, and multi-step workflows across different models.
LLM-as-judge scoring is a validation mechanism where a language model evaluates and scores agent responses against defined criteria. It automates the assessment of response quality within your test cases to ensure outputs meet expected standards.
Yes, you can run evaluations across a matrix of different LLM models. Multi-model testing allows you to validate agent consistency and quality using the same YAML test cases across various model configurations.
You verify tool selection and skill invocation using defined assertion types within your YAML test cases. These assertions check whether the agent correctly identifies and triggers the appropriate tools and skills for specific user requests.
Yes, the evaluation framework includes mock service definitions. These mocks simulate external interactions, allowing you to isolate and test multi-step agent workflows and tool selection logic without relying on live services.
VjayRamDefine DSPy signatures and modules with deterministic evaluation across splits.
coval-aiAnalyze human versus LLM judge annotation discrepancies to refine evaluation prompts.
kk20300113-pngConfigure autonomous agent loops with quality gates and recovery procedures.
DhumitechAutomate AI agent evaluation and benchmarking with repeatable test pipelines.
