infra-opentelemetry-instrumentation

What problem does it solve?

This Skill helps you instrument data engineering pipelines so you can see end-to-end latency, pinpoint where time is spent, and correlate traces across Airflow, Spark, dbt, and Kafka.

Core Features & Use Cases

• Zero-code Python auto-instrumentation: bootstrap OpenTelemetry instrumentation for common libraries (HTTP, SQLAlchemy, Kafka clients, and more) without manual code changes.
• Manual spans for ETL phases: create and annotate spans for extract/transform/load so row counts, partition dates, and target tables become searchable telemetry.
• Context propagation across services: propagate trace context and baggage (e.g., partition date, run id) from Airflow tasks through downstream tasks and via Kafka headers.
• OTLP exporter + Collector pipeline tuning: route telemetry to an OpenTelemetry Collector and apply tail sampling (keep errors and slow traces) to control cost.
• Trace/log correlation: inject trace_id and span_id into logs to link log events back to the corresponding trace in backends like Grafana Tempo.
• Use Case: When an ETL run is slow or failing, follow one pipeline run through Airflow task execution, into Spark/dbt-related spans, and across Kafka messaging to identify the exact bottleneck and affected partitions.

Quick Start

Instrument your ETL Python code by setting OTEL_SERVICE_NAME and OTLP exporter environment variables, then run your pipeline using opentelemetry-instrument so traces are exported to your OpenTelemetry Collector.