skills.rest
PostHog
PostHogOFFICIAL
Explore all skills by this author →
Website𝕏@PostHogGitHub

signals-scout-mcp-tool-calls

Detects failing, slow, or confusing MCP tools from $mcp_tool_call telemetry and files per-category reports.

713|118|Updated Aug 11, 2020
Search Tags:#error detection#mcp#PostHog#telemetry analysis#HogQL#tool quality#signals scout
One-click install
npx skills add https://github.com/PostHog/posthog-foss --skill signals-scout-mcp-tool-calls
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: signals-scout-mcp-tool-calls
Source: https://github.com/PostHog/posthog-foss/tree/main/products/signals/skills/signals-scout-mcp-tool-calls
Command: npx skills add https://github.com/PostHog/posthog-foss --skill signals-scout-mcp-tool-calls

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

PostHog MCP tool quality issues hide inside raw $mcp_tool_call telemetry: high failure rates, agents hammering tools with retries, slow responses, and context-bloating outputs. This Skill analyzes that telemetry, groups problem tools by owning product category, and files actionable reports with fix hypotheses so teams know exactly which tools need improvement.

Core Features & Use Cases

  • Multi-lens detection: Ranks tools by failure rate, per-session struggle/retry patterns, p95 latency, error classes, and response bloat using a HogQL query cookbook over $mcp_tool_call events.
  • Regime-aware analysis: Probes field coverage first to distinguish PostHog's hono server data from external customer SDK data, then adapts which diagnostic lenses are usable.
  • Category-grained reporting: Files or edits one inbox report per product category with per-tool fix hypotheses, dedupe via scratchpad memory, reviewer routing, and a fix-loop metric for autonomous implementation tasks.
  • Use Case: A daily scheduled run finds that the data-warehouse category has four tools failing at 40% across 30 users; it edits the existing category report with fresh numbers and a schema-fix hypothesis routed to the owning team.

Quick Start

Run the MCP tool-quality scout to analyze this project's $mcp_tool_call events from the last 7 days and file reports for any tool categories that need improvement.

Frequently Asked Questions about signals-scout-mcp-tool-calls

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I detect failing MCP tools in PostHog analytics?▼

Query the $mcp_tool_call event with the failure leaderboard HogQL: group by effective tool name, compute error rate from $mcp_is_error, and filter by volume and reach floors. Tools with high rates across many users and sessions are report candidates.

How to measure MCP tool quality beyond error rates?▼

Track per-session struggle (agents calling a tool 3+ times or failing then retrying), p95 $mcp_duration_ms latency, and output token bloat. These signals reveal confusing schemas and slow tools even when error rates look healthy.

What is the difference between hono and external-SDK MCP telemetry?▼

PostHog's hono server captures $mcp_error_type, $mcp_tool_category, mode, and token counts but omits error messages. External customer SDK servers capture $mcp_error_message but no error classes or categories. Run the field-coverage probe first to determine which regime applies.

Why does $mcp_error_type return inconsistent counts in HogQL?▼

Bare value equality on $mcp_error_type gives contradictory counts across query shapes. Define classified failures with a positive toString() IN whitelist of known classes and compute unclassified failures by subtraction, never with NOT IN.

When should an MCP tool problem not be reported?▼

Skip single-user or single-session failures, low absolute volume, self-recovering validation retries, rate limiting alone, errors during known PostHog incidents, and structurally slow tools like large exports. Category-level sums of individually healthy tools are also not findings.

Related Skills

Explore More Data & Analytics Skills→
PostHogPostHog
58

integration-javascript_node

Instrument Node.js applications with PostHog to capture events and identify users.

Official
Advanced
zxdxjtuzxdxjtu
8

eval-fix

Iteratively generate and evaluate fixes until feature gaps converge.

Community
Advanced
KhoiwallKhoiwall

deep-research

Guide memory-first analysis, LSP exploration, and pattern discovery across codebases.

Community
Advanced
byaan-aibyaan-ai
94

posthog

Query events and analytics via HogQL using the PostHog REST API.

Official
Advanced