skills.rest
PostHog
PostHogOFFICIAL
Explore all skills by this author →
Website𝕏@PostHogGitHub

improving-mcp-tools

Runs eval-backed iterations that measure, fix, and validate MCP tool improvements.

713|118|Updated Aug 11, 2020
Search Tags:#mcp#benchmark#regression testing#campaign#eval harness#tool descriptions#tool discoverability
One-click install
npx skills add https://github.com/PostHog/posthog-foss --skill improving-mcp-tools
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: improving-mcp-tools
Source: https://github.com/PostHog/posthog-foss/tree/main/products/mcp_analytics/skills/improving-mcp-tools
Command: npx skills add https://github.com/PostHog/posthog-foss --skill improving-mcp-tools

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

MCP tool quality issues like poor discoverability, confusing descriptions, and schema errors are hard to fix without evidence. This Skill provides a disciplined campaign loop that measures the agent experience with an eval harness, picks the highest-impact problem from production data, makes one bounded fix, and ships it only if before/after scores improve.

Core Features & Use Cases

  • Eval-driven measurement: Run probe mode (deterministic tool-presence and latency checks) and agent mode (LLM replay with task success and tool-selection scoring) against a seeded local stack.
  • Evidence-based issue selection: Pull production signals with MCP analytics queries (tool stats, failures, descriptions, sample intents) and rank issues by reach times severity.
  • Guardrailed shipping: Enforce a strict file allowlist, one issue per iteration, mandatory before/after benchmark evidence in every PR, and a campaign journal so later iterations can resume.
  • Use Case: An agent notices agents frequently pass sql instead of query to the execute-sql tool. It baselines the benchmark, clarifies the input description in tools.yaml, re-runs the harness, confirms task success improves from 19/27 to 22/27, and opens a draft PR with the eval evidence block.

Quick Start

Ask the agent to run an improve-my-MCP campaign against the local MCP stack and fix the highest-impact tool problem with eval evidence.

Frequently Asked Questions about improving-mcp-tools

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I improve MCP tool discoverability with eval evidence?▼

Run the eval harness in agent mode to get a baseline of task success and tool-selection accuracy, then rewrite the tool description and re-run the same benchmark. Ship the change only if scores improve and a no-regression sample stays unchanged.

How to run the MCP eval harness in probe mode?▼

From services/mcp, run the probe script with LIVE_MCP_URL and LIVE_MCP_TOKEN set, pointing at a seeded local or devbox stack. It reports tool-presence misses, probe failures, and latency p50/p95, exiting non-zero on regression.

What files can an MCP improvement campaign PR change?▼

Campaign PRs may only touch productsskills, services/mcp/evals, codegen outputs of the tool generators, and docs. Handler code, manifests, workflows, and migrations are handed to a human instead.

Why did my MCP description change hurt another tool's traffic?▼

A description that helps one intent can steal traffic from an overlapping tool. Snapshot the intent-cluster tool_overlaps before a rewrite, recompute after, and treat a capture shift in an overlapping pair as a regression signal.

When should a campaign change be discarded instead of shipped?▼

Discard the change when the target metric does not improve or any other metric degrades in the no-regression sample. Journal the discarded attempt, and park the issue permanently after two failed attempts.

Related Skills

Explore More Software Engineering Skills→
messarimessari
6

messari-crypto

Queries REST API and MCP data stack for crypto market metrics, signals, news, research, and tokenomics.

Official
Basic
gaogg521gaogg521
33

pr-automation

Automate pull request reviews, fixes, CI checks, and merges.

Community
Advanced
VladSezVladSez
1.1k

keyword-clustering

Cluster keywords by search intent and map them to existing or proposed pages.

Community
Intermediate
tdsporatdspora
18

syngen-harness-update

Require human approval for harness file updates through a structured review workflow.

Community
Advanced