skills.rest
jmrplens
jmrplensCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

create-mcp-evaluation

Creates verified question-answer pairs to evaluate MCP server tool quality.

37|4|Updated Apr 11, 2026
Search Tags:#benchmarking#evaluation#mcp#question generation#gitlab#llm testing#tool quality
One-click install
npx skills add https://github.com/jmrplens/gitlab-mcp-server --skill create-mcp-evaluation-jmrplens
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: create-mcp-evaluation
Source: https://github.com/jmrplens/gitlab-mcp-server/tree/main/.github/skills/create-mcp-evaluation
Command: npx skills add https://github.com/jmrplens/gitlab-mcp-server --skill create-mcp-evaluation-jmrplens

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Measuring whether an MCP server actually helps LLMs accomplish real tasks is hard without a structured benchmark. This Skill guides the creation of realistic, multi-hop evaluation questions with verifiable answers that test tool design, schemas, and descriptions. ## Core Features & Use Cases - Structured Evaluation Design: Walks through tool inspection, read-only data exploration, question design, and manual answer verification. - Quality Constraints: Enforces independence, read-only operations, stable single-value answers, and multi-tool-call complexity across diverse answer types. - Repository Integration: Targets the test/e2e/modeleval harness with typed corpus cases, a no-stimulus-leak contract, and a fake:perfect rehearsal mode that costs nothing. - Use Case: After shipping a GitLab MCP server, generate 10 cross-resource questions (issues, MRs, pipelines, users), verify each answer with the real tools, then run them against a model with a budget ceiling. ## Quick Start Ask the assistant to create ten read-only evaluation Q&A pairs for the MCP server and verify each answer using only its tools.

Frequently Asked Questions about create-mcp-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate the quality of an MCP server?▼

Create realistic questions that require multiple read-only tool calls, verify each answer yourself using only the server's tools, then run the corpus against a model. This Skill guides that process and outputs XML qa_pair entries.

What makes a good MCP evaluation question?▼

Good questions are independent, read-only, multi-hop, and have a single stable verifiable answer. They should require chaining several tool calls and avoid naming the tool or parameters of their own answer key.

How do I run MCP evaluations without spending money on models?▼

Use the fake:perfect rehearsal mode, which replays each case's answer key through the full pipeline without calling any provider. Run MODELEVAL_MODELS=fake:perfect make modeleval-ce to validate the harness for free.

Why does the evaluation reject prompts that name their own answer?▼

A prompt containing its case's tool, action, or parameter names measures copying rather than tool selection. The TestContract_NoStimulusNamesItsOwnAnswer check fails such cases unless the literal is explicitly declared with a reason.

Can evaluation questions use write operations on the MCP server?▼

No. All questions must use only read-only, non-destructive operations so answers stay stable and the evaluation can run repeatedly against live data without side effects.

Related Skills

Explore More Software Engineering Skills→
midas44midas44

postgres-db

Query PostgreSQL databases across QA1, TM, Stage via read-only MCP servers.

Community
Intermediate
RohithdgrrRohithdgrr

agent-designer

Design multi-agent architectures, generate tool schemas, and evaluate agent execution logs.

Community
Advanced
aufrankaufrank

tool-quality-audit

Audits tool and MCP server integrations for deterministic behavior, error contracts, and schema stability.

Community
Intermediate
zilliztechzilliztech
3

rag

Ground LLM answers in document corpora with cited source chunks.

Official
Advanced