debug-issue-with-datadog

Diagnose production issues by correlating Datadog telemetry with Langfuse source code.

34.0k|3.7k|Updated May 18, 2023
One-click install
npx skills add https://github.com/langfuse/langfuse --skill debug-issue-with-datadog
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: debug-issue-with-datadog
Source: https://github.com/langfuse/langfuse/tree/main/.agents/skills/debug-issue-with-datadog
Command: npx skills add https://github.com/langfuse/langfuse --skill debug-issue-with-datadog

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Investigating a production bug report, Linear ticket, or incident requires manually stitching together Datadog spans, logs, metrics, and the Langfuse codebase. This Skill structures that investigation so root causes are established from telemetry evidence rather than guesswork.

Core Features & Use Cases

  • Structured intake: Extracts subsystem, region, time window, project IDs, and error fragments from Linear issues, GitHub issues, or pasted error reports before querying.
  • Broad Datadog sweep: Runs APM span, log, metric, and monitor queries across prod-eu and prod-us, then clusters errors by projectId and message to identify distinct root causes.
  • Repo debug map: Maps subsystems like PostHog integration, blob storage export, and evaluation execution to exact queue files, handler directories, and span resource names.
  • Use Case: Given a Linear issue reporting PostHog integration failures, the Skill pulls the issue context, sweeps Datadog across both regions, clusters errors by tenant, maps each cluster to handler files in worker/src/features, and produces a prioritized P0/P1/P2 patch analysis.

Quick Start

Investigate Linear issue LFE-9475 using Datadog telemetry and produce a root cause analysis with suggested patches.

Frequently Asked Questions about debug-issue-with-datadog

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I investigate a production issue using Datadog and the Langfuse repo?

Start with intake: fetch the Linear or GitHub issue, extract the subsystem, region, time window, and project IDs. Then run a broad Datadog sweep of APM spans, logs, and metrics across both EU and US, cluster errors by projectId and message, and map each cluster to handler files in the repo.

How do I find which Langfuse worker queue is failing in Datadog?

Worker async jobs emit spans named 'process <queue-name>' via instrumentAsync wrappers. The repo debug map lists each subsystem's queue file, handler directory, and span resource name, so you can filter APM with service:worker and the matching resource_name.

Does this Skill work for both EU and US Langfuse deployments?

Yes. Two Datadog MCP servers are used, one for datadoghq.eu and one for datadoghq.com, and queries run against both unless intake clearly localizes the incident. Regional disparity between prod-eu and prod-us is checked first since it often rules hypotheses in or out.

When should I not use Datadog-based issue debugging?

Do not use it when the task is implementing a known fix rather than investigating what is broken. Implementation tasks should go to backend development guidelines or the relevant package guide instead of an investigative telemetry sweep.

Why does a Datadog query return no results for a known error?

Common causes are a wrong case-sensitive resource name, a time window that misses when the issue fired, or querying the wrong regional MCP server. Tenant tags may also live on the api-auth-verify child span rather than the request root span, requiring correlation by trace ID.