skills.rest
ivanshamaev
ivanshamaevCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

dataops-airflow-observability

Export Apache Airflow scheduler, task, and DAG signals to metrics, traces, and logs.

14|1|Updated May 5, 2026
Search Tags:#observability#grafana#prometheus#alerting#structured logging#open telemetry#apache airflow
One-click install
npx skills add https://github.com/ivanshamaev/de-agent-skills --skill dataops-airflow-observability
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: dataops-airflow-observability
Source: https://github.com/ivanshamaev/de-agent-skills/tree/main/group_skills/infra_dataops_group_skills/dataops_airflow_observability
Command: npx skills add https://github.com/ivanshamaev/de-agent-skills --skill dataops-airflow-observability

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill solves the problem of operating Apache Airflow “blindly” by turning scheduler health, task performance, queueing, and SLA misses into actionable metrics, dashboards, traces, and alerting signals.

Core Features & Use Cases

  • Metrics collection for Airflow: Configure StatsD-to-Prometheus and/or OpenTelemetry to export scheduler heartbeat, task duration/failures, DAG parse time, and executor queue/running counts.
  • Observability UI and alerting: Build Grafana panels for success/failure rate, task duration percentiles, pool slot utilization, and SLA miss detection; implement Prometheus alert rules for stale scheduler heartbeats, failure spikes, pool starvation, DAG parse errors, and SLA misses.
  • Operational diagnostics with context: Enable structured JSON task logging for easier log correlation and add OpenTelemetry traces to follow task spans across the execution chain.

Quick Start

Set up Airflow metrics with StatsD or OpenTelemetry, then create the Grafana dashboard and Prometheus alert rules to monitor DAG success/failure, task duration (including P95), pool slot utilization, scheduler heartbeat staleness, and DAG parsing/import errors.

Frequently Asked Questions about dataops-airflow-observability

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up Apache Airflow observability with Prometheus and Grafana?▼

To monitor Airflow task duration and failures, configure StatsD-to-Prometheus or OpenTelemetry export in airflow.cfg to emit task duration, failure counts, and DAG parse time metrics for Grafana dashboards.

What Prometheus alert rules should I define for Apache Airflow?▼

Prometheus alert rules for Airflow should target stale scheduler heartbeats, failure spikes, pool slot starvation, DAG parse errors, and SLA misses to ensure production pipeline reliability.

How do I track Airflow SLA misses and pool utilization in Grafana?▼

Tracking Airflow SLA misses and pool utilization in Grafana requires exporting scheduler and executor signals via StatsD or OpenTelemetry, then building panels for SLA miss detection and pool slot utilization metrics.

Can I use OpenTelemetry traces with Apache Airflow for task diagnostics?▼

OpenTelemetry traces can be used with Apache Airflow by enabling trace instrumentation to follow task spans across the execution chain, combined with structured JSON logging for easier log correlation during incident diagnosis.

What is the best way to diagnose slow or failing Airflow DAGs in production?▼

Diagnosing slow or failing Airflow DAGs requires exporting DAG lifecycle signals to metrics and logs, using Grafana panels for task duration percentiles, and enabling structured JSON logging for log correlation.

Does Apache Airflow support structured JSON logging for log correlation?▼

Apache Airflow supports structured JSON logging to enable easier log correlation during operational diagnostics, which pairs with OpenTelemetry traces to follow task spans across the execution chain.

Related Skills

Explore More Software Engineering Skills→
PostHogPostHog
58

logs-nextjs

Configure Next.js 15.3+ apps to emit OpenTelemetry logs to PostHog.

Official
Advanced
BagelHoleBagelHole
46

alerting-oncall

Configure Prometheus alert rules and on-call rotations with PagerDuty or Grafana OnCall.

Community
Advanced
saahilvasdevsaahilvasdev

grafana-dashboards

Design and deploy Grafana dashboards for Prometheus metrics with YAML frontmatter.

Community
Advanced
doanchienthangdevdoanchienthangdev
4

distributed-tracing

Implement distributed tracing with OpenTelemetry, Jaeger, and Zipkin for microservices.

Community
Advanced