tempo-troubleshoot

Guide TraceQL query crafting for diagnosing latency and errors in Grafana Tempo distributed traces.

1|Updated Dec 4, 2025
One-click install
npx skills add https://github.com/timbuchinger/loadout --skill tempo-troubleshoot
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: tempo-troubleshoot
Source: https://github.com/timbuchinger/loadout/tree/main/skills/tempo-troubleshoot
Command: npx skills add https://github.com/timbuchinger/loadout --skill tempo-troubleshoot

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill simplifies the complex task of troubleshooting distributed traces and crafting effective TraceQL queries in Grafana Tempo. It helps you quickly identify latency, errors, and bottlenecks across your services.

Core Features & Use Cases

  • Efficient TraceQL Query Building: Guides in constructing precise TraceQL queries to filter traces by service, status, duration, and other attributes.
  • Distributed Trace Debugging: Helps you navigate and understand complex distributed traces for request-level root cause analysis.
  • Log & Metric Correlation: Provides best practices for linking traces with relevant logs (Loki) and metrics (Mimir) for a holistic view.
  • Use Case: You're investigating a slow API response. This Skill helps you craft a TraceQL query to find all traces for the 'payments' service that took longer than 1 second, allowing you to pinpoint the exact span causing the delay.

Quick Start

Craft a TraceQL query to find all error traces in the 'payments' service from the last 15 minutes, filtering by service name.

Frequently Asked Questions about tempo-troubleshoot

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I debug slow traces in Tempo using TraceQL?

TraceQL lets you filter Tempo traces by service, duration, and status to pinpoint latency. Construct queries scoped to a specific service and time range, then correlate results with Loki logs and Mimir metrics for root-cause analysis across microservices.

What's the best way to craft a TraceQL query for error traces?

Build TraceQL queries by filtering on service name, error status, and time range to isolate failures. Avoid high-cardinality attributes that degrade performance, and link matching traces to logs and metrics for deeper context.

Can I correlate distributed traces with logs and metrics?

Yes. Use TraceQL to find traces, then link them to Loki logs and Mimir metrics within the same service and time window. This trace-log-metrics correlation provides holistic visibility into request-level failures and bottlenecks.

How do I perform root-cause analysis across microservices in Tempo?

Navigate distributed traces to examine spans across services, identify which span caused latency or errors, and correlate with logs and metrics. Scope queries to your service and time range to keep analysis focused and performant.

What are the limitations when building TraceQL queries?

TraceQL does not execute queries directly in this Skill; it provides guidance only. Avoid high-cardinality attributes, scope queries tightly by service and time range, and use external Tempo or Grafana instances to run the queries you construct.

When should I use distributed tracing to investigate payment errors?

Use distributed tracing when errors span multiple services and logs alone don't show the flow. TraceQL queries on the payments service reveal which span failed, then correlate with logs and metrics to identify root cause.