skills.rest
genmcp
genmcpOFFICIAL
Explore all skills by this author →
GitHub

Create MCP Eval

Create or modify MCP server evaluation YAMLs with task sets and assertions.

24|26|Updated Sep 26, 2025
Search Tags:#automation#testing#evaluation#configuration#mcp#ai agent#kubernetes
One-click install
npx skills add https://github.com/genmcp/gevals --skill create-mcp-eval
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: Create MCP Eval
Source: https://github.com/genmcp/gevals/tree/main/.claude/skills/create-eval
Command: npx skills add https://github.com/genmcp/gevals --skill create-mcp-eval

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Setting up and configuring evaluations for Model Context Protocol (MCP) servers can be complex and time-consuming, requiring multiple configuration files and scripts. This Skill streamlines the process, allowing you to quickly define and manage comprehensive test suites for your AI agents and MCP tools.

Core Features & Use Cases

  • Unified Eval Creation: Generate and manage all necessary configuration files (eval.yaml, agent.yaml, mcp-config.yaml, task files) from a single, guided interface.
  • Automated Testing Setup: Quickly configure robust tests for your MCP server's tools, ensuring they are discoverable, well-described, and correctly implemented.
  • Use Case: You've developed a new set of tools for your Kubernetes MCP server. Use this Skill to define a comprehensive evaluation suite that tests tool discoverability, schema correctness, and agent interaction, all with minimal manual configuration, ensuring your AI agents can effectively use your tools.

Quick Start

After this Skill helps you create your evaluation files,

run your tests using the 'gevals' command:

gevals run <path to eval yaml file>

If the 'gevals' binary is not in your system's PATH,

you may need to specify its full path or ask the user for its location.

Frequently Asked Questions about Create MCP Eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up evaluation tests for MCP servers?▼

Create MCP Eval automates evaluation setup by generating configuration files (eval.yaml, agent.yaml, mcp-config.yaml, task files) in a guided interface. It composes tasks, MCP config, agent definitions, and eval references to define comprehensive test suites for your MCP server tools.

What goes into an MCP evaluation YAML file?▼

An eval YAML requires kind, metadata, config.agent, mcpConfigFile, and taskSets (path or glob). Optional assertions include toolsUsed, minToolCalls, maxToolCalls, prompts, resources, and callOrder to specify functional and technical test requirements.

Can I test tool discoverability and schema correctness for Kubernetes MCP servers?▼

Yes. Create MCP Eval lets you define evaluation suites that verify tool discoverability, schema correctness, and agent interaction for Kubernetes and other MCP servers, ensuring AI agents can effectively use your tools.

Do I need to write evaluation configurations manually for each MCP tool?▼

No. This Skill streamlines the process by automating configuration file generation and management, allowing you to quickly define and manage test suites without manual setup for multiple MCP tools and servers.

How do I run tests after creating MCP evaluations?▼

After Create MCP Eval generates your eval.yaml file, run tests using the gevals command: `gevals run <path to eval yaml file>`. If gevals is not in your PATH, specify its full path.

What's the difference between defining task assertions for MCP evaluations?▼

Assertions control test behavior: toolsUsed specifies which tools to invoke, minToolCalls and maxToolCalls set call thresholds, prompts define test scenarios, resources specify required context, and callOrder validates interaction sequence.

Related Skills

Explore More Software Engineering Skills→
AradotsoAradotso
7

ktx-ai-data-agents

Build an agent context layer from metrics and wiki definitions for data warehouse queries.

Official
Advanced
iDako7iDako7

promptfoo-evals

Create and update promptfoo evaluation suites for prompts, providers, tests, and assertions.

Community
Intermediate
RamVegirajuRamVegiraju
4

modify-agent

Modify Databricks agent code and configurations for tailored behavior.

Community
Advanced
pradyumnasagarpradyumnasagar
6

roadmap-author

Guide authors through creating and validating ORS SKILL.md files.

Community
Intermediate