build-evals

Generate LLM evaluation tests for MCP tool collections from discovery manifests.

2|3|Updated Jan 15, 2026

Search Tags:#automation #testing #code generation #llm #evaluation #mcp

Ready to run

npx skills add https://github.com/umbraco/Umbraco-MCP-Base --skill build-evals

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: build-evals
Source: https://github.com/umbraco/Umbraco-MCP-Base/tree/main/plugins/skills/build-evals
Command: npx skills add https://github.com/umbraco/Umbraco-MCP-Base --skill build-evals

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill automates the creation of LLM evaluation tests for your MCP tool collections, ensuring robust and reliable AI agent behavior.

Core Features & Use Cases

Automated Test Generation: Creates eval setup and scenario test files based on .discover.json and existing tool definitions.
Workflow Testing: Designs and implements tests for CRUD, read-only, search, and hierarchical workflows.
Use Case: After defining your Umbraco API tools, use this skill to automatically generate comprehensive LLM evaluation tests, significantly speeding up your development and testing cycle.

Quick Start

Use the build-evals skill to generate LLM eval tests for all MCP tool collections.

Frequently Asked Questions about build-evals

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I generate LLM evaluation tests for MCP tool collections?▼

This Skill automates LLM evaluation test generation by reading discovery manifests and tool definitions to create eval setup and scenario test files for your MCP tool collections.

What types of workflows do LLM eval tests cover for MCP tools?▼

LLM eval tests for MCP tools cover CRUD, read-only, search, and hierarchical workflows, ensuring robust and reliable AI agent behavior across different interaction patterns.

Do I need an LLM API key to run generated eval tests for MCP tools?▼

Yes, executing the generated LLM eval tests requires an LLM API key and a built project to validate your MCP tool collections against the designed scenarios.

Can I automate eval scenario creation from .discover.json files?▼

Yes, automated test generation creates eval setup and scenario test files directly based on .discover.json files and existing tool definitions within your project structure.

What are the limitations of automating LLM evaluation test generation?▼

Automated LLM evaluation test generation requires the project to be built beforehand and depends on an active LLM API key for execution, meaning it cannot run in isolated or unbuilt environments.