dataops-airflow-observability

What problem does it solve?

This Skill solves the problem of operating Apache Airflow “blindly” by turning scheduler health, task performance, queueing, and SLA misses into actionable metrics, dashboards, traces, and alerting signals.

Core Features & Use Cases

• Metrics collection for Airflow: Configure StatsD-to-Prometheus and/or OpenTelemetry to export scheduler heartbeat, task duration/failures, DAG parse time, and executor queue/running counts.
• Observability UI and alerting: Build Grafana panels for success/failure rate, task duration percentiles, pool slot utilization, and SLA miss detection; implement Prometheus alert rules for stale scheduler heartbeats, failure spikes, pool starvation, DAG parse errors, and SLA misses.
• Operational diagnostics with context: Enable structured JSON task logging for easier log correlation and add OpenTelemetry traces to follow task spans across the execution chain.

Quick Start

Set up Airflow metrics with StatsD or OpenTelemetry, then create the Grafana dashboard and Prometheus alert rules to monitor DAG success/failure, task duration (including P95), pool slot utilization, scheduler heartbeat staleness, and DAG parsing/import errors.