skills.rest
MaterializeInc
MaterializeIncOFFICIAL
Explore all skills by this author →
Website𝕏@MaterializeIncGitHub

mz-query-tracing

Diagnose SQL query latency in Materialize using OpenTelemetry distributed traces from Tempo.

6.4k|512|Updated Feb 22, 2019
Search Tags:#query performance#distributed tracing#tempo#openTelemetry#materialize#span analysis#latency debugging
One-click install
npx skills add https://github.com/MaterializeInc/materialize --skill mz-query-tracing
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: mz-query-tracing
Source: https://github.com/MaterializeInc/materialize/tree/main/.agents/skills/mz-query-tracing
Command: npx skills add https://github.com/MaterializeInc/materialize --skill mz-query-tracing

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

When a SQL statement in Materialize runs slowly, it is hard to know where the time is actually spent. This Skill captures distributed traces of query execution via OpenTelemetry and Tempo, then breaks down span durations and self-times so you can pinpoint the exact code paths causing latency.

Core Features & Use Cases

  • Trace Capture Workflow: Guides starting the Tempo/Grafana monitoring stack, launching environmentd with tracing enabled, and setting the opentelemetry_filter at runtime.
  • Trace ID Extraction: Uses psql with emit_trace_id_notice to capture the trace ID for any SQL statement.
  • Span Tree Analysis: The trace_tree.py script parses Tempo's OTLP JSON output, decodes base64 span IDs, and prints a hierarchical span tree with self-time rankings and source code locations.
  • Use Case: A developer notices a CREATE TABLE statement takes several seconds. They trace it, fetch the trace from Tempo, and discover most time is spent in group_commit_apply::append_fut writing to the durable log.

Quick Start

Trace my slow SQL query in my local Materialize instance and show me where the execution time is going.

Frequently Asked Questions about mz-query-tracing

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I trace a slow SQL query in Materialize?▼

Run the query via psql with emit_trace_id_notice enabled to capture a trace ID, wait about 10 seconds for spans to flush, then fetch the trace from the Tempo API on port 3200 and analyze it with the trace_tree.py script.

How do I analyze Tempo trace JSON for span latency?▼

Use the trace_tree.py script, which parses Tempo's OTLP JSON format, decodes base64 span IDs, builds a parent-child span tree, and prints the top 15 spans by self-time plus a hierarchical tree with source code locations.

Why are my Materialize traces empty or missing in Tempo?▼

Traces are lost if Tempo was not running before environmentd started, since spans emitted early are silently dropped. Also check that opentelemetry_filter is set to debug or trace, and retry fetching because the default batch export delay is 5 seconds.

Should I trace debug or optimized builds of Materialize?▼

Always trace optimized builds using bin/environmentd --optimized --monitoring. Debug builds inflate self-times with unoptimized code and extra debug checks, producing misleading latency breakdowns that do not reflect production behavior.

How long does Tempo retain traces?▼

Tempo retains traces for only 15 minutes by default, as configured in misc/monitoring/tempo.yml. Fetch traces promptly after running your query, or view them in Grafana at localhost:3000 via the Tempo datasource.

Related Skills

Explore More Software Engineering Skills→
timequitytimequity
6

monitoring-setup

Set up Prometheus, Grafana, Loki, and Jaeger for Kubernetes observability.

Official
Advanced
software-mansionsoftware-mansion
1.9k

argent-react-native-profiler

Profile React Native Hermes commits and CPU time to rank re-renders.

Official
Advanced
williamzujkowskiwilliamzujkowski
6

Observability Unified Dashboard

Designs unified observability dashboards correlating metrics, logs, and traces across Prometheus, Loki, Tempo, and Grafana 11.

Community
Advanced
israel-salgadoisrael-salgado
2

dt-obs-tracing

Query Dynatrace distributed traces and spans with DQL.

Community
Advanced