infra-opentelemetry-instrumentation

Instrument Python ETL pipelines with OpenTelemetry and export traces via OTLP.

14|1|Updated May 5, 2026

Search Tags:#tracing #airflow #kafka #opentelemetry #grafana tempo #otlp exporter #etl observability

One-click install

npx skills add https://github.com/ivanshamaev/de-agent-skills --skill infra-opentelemetry-instrumentation

Or copy as Structured Prompt for Agent▼

Please help me install this Agent Skill.
Skill: infra-opentelemetry-instrumentation
Source: https://github.com/ivanshamaev/de-agent-skills/tree/main/group_skills/infra_dataops_group_skills/infra_opentelemetry_instrumentation
Command: npx skills add https://github.com/ivanshamaev/de-agent-skills --skill infra-opentelemetry-instrumentation

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps you instrument data engineering pipelines so you can see end-to-end latency, pinpoint where time is spent, and correlate traces across Airflow, Spark, dbt, and Kafka.

Core Features & Use Cases

Zero-code Python auto-instrumentation: bootstrap OpenTelemetry instrumentation for common libraries (HTTP, SQLAlchemy, Kafka clients, and more) without manual code changes.
Manual spans for ETL phases: create and annotate spans for extract/transform/load so row counts, partition dates, and target tables become searchable telemetry.
Context propagation across services: propagate trace context and baggage (e.g., partition date, run id) from Airflow tasks through downstream tasks and via Kafka headers.
OTLP exporter + Collector pipeline tuning: route telemetry to an OpenTelemetry Collector and apply tail sampling (keep errors and slow traces) to control cost.
Trace/log correlation: inject trace_id and span_id into logs to link log events back to the corresponding trace in backends like Grafana Tempo.
Use Case: When an ETL run is slow or failing, follow one pipeline run through Airflow task execution, into Spark/dbt-related spans, and across Kafka messaging to identify the exact bottleneck and affected partitions.

Quick Start

Instrument your ETL Python code by setting OTEL_SERVICE_NAME and OTLP exporter environment variables, then run your pipeline using opentelemetry-instrument so traces are exported to your OpenTelemetry Collector.

Frequently Asked Questions about infra-opentelemetry-instrumentation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I trace ETL pipeline latency across Airflow tasks and Kafka messages?▼

Yes, Python auto-instrumentation bootstraps OpenTelemetry for common libraries like HTTP, SQLAlchemy, and Kafka clients without requiring manual code changes to your data pipelines.

How do I add searchable telemetry data to my extract, transform, and load phases?▼

You create and annotate manual OpenTelemetry spans for your extract, transform, and load phases so row counts, partition dates, and target tables become searchable telemetry attributes.

How do I control observability costs when exporting OpenTelemetry traces?▼

You control observability costs by routing telemetry via the OTLP exporter to an OpenTelemetry Collector and applying tail sampling to keep only errors and slow traces.

Can I correlate my application logs with OpenTelemetry traces in Grafana Tempo?▼

Yes, you can inject trace_id and span_id into your application logs to link log events back to their corresponding traces in backends like Grafana Tempo for easier debugging.

Do I need an OpenTelemetry Collector to use this instrumentation for my data platform?▼

Yes, an OpenTelemetry Collector is required to receive OTLP exported telemetry, apply tail sampling, and route the observability data to your configured backend.