Manage agent evaluations in Relevance AI with test cases and eval runs.
npx skills add https://github.com/RelevanceAI/agent-skills --skill relevance-evals
Please help me install this Agent Skill. Skill: relevance-evals Source: https://github.com/RelevanceAI/agent-skills/tree/main/reference/relevance-evals Command: npx skills add https://github.com/RelevanceAI/agent-skills --skill relevance-evals
Manages the end-to-end evaluation process for AI agents, enabling structured test design, execution, and results analysis within Relevance AI.
Create a new test set, add test cases, run an evaluation batch, and review the results.
High-intent search queries and answers about installing and using this skill.
To automate agent evaluations, you can create test sets, add test cases, run evaluation batches, and review results. This workflow supports generating conversations and scoring them using an LLM judge.
The generate_and_score mode runs conversations and evaluates them, while score_only evaluates existing outputs. Both modes provide batch-based result tracking with per-run rule judgments and summaries for agent testing.
You can manage test cases by creating a new test set and adding specific scenarios. The system provides guidelines for designing reliable, observable rules to ensure accurate end-to-end agent evaluations.
Yes, you can review batch-level summaries and per-run rule judgments. The evaluation workflow allows you to inspect individual results after triggering batches to analyze agent performance thoroughly.
The best way to design reliable rules is to follow structured guidelines for creating observable scenarios. This ensures your test cases accurately measure agent performance during generate_and_score or score_only eval runs.
yu13130122297Identify uncertainties in research framing, datasets, metrics, and baselines.
npowAnalyze agent session failures and convert verified behavioral problems into enforcement patches.
microsoftGenerates categorized evaluation test sets for Microsoft Copilot Studio agents.
abdullah1854Develop evaluation frameworks for non-deterministic agent systems.
