dataops-airflow-production-readiness

What problem does it solve?

This Skill helps you prepare an Apache Airflow deployment so production DAGs run reliably, avoid scheduler/metadata-database overload, and remain safe to re-execute without corrupting data.

Core Features & Use Cases

• Idempotent task design: choose UPSERT or INSERT OVERWRITE patterns (including partition overwrite) so reruns don’t duplicate rows or produce inconsistent outputs.
• DAG parse-time safety: prevent anti-patterns like Variable.get() at module/DAG parse time and keep heavy imports out of the global scope.
• Operational production readiness: configure retries with exponential backoff, SLA callbacks, pools for throttling, KubernetesExecutor resources, metadata DB maintenance, and health checks.
• Use Case: audit a newly added ETL DAG (e.g., orders -> bronze -> silver) and ensure each task can be safely retried, scales on Kubernetes, and alerts correctly when SLAs are missed.

Quick Start

Ask the AI to review your Airflow DAG code for production anti-patterns and provide concrete fixes for idempotency, parse-time database hits, retries/SLA, pools, KubernetesExecutor settings, and metadata DB maintenance.