What problem does it solve? Building reliable data pipelines in Apache Airflow requires knowing the right patterns for DAG design, task dependencies, error handling, and testing. This Skill provides production-ready templates and best practices so you avoid common pitfalls like non-idempotent tasks, missing retries, and untested DAGs. ## Core Features & Use Cases - DAG Design Patterns: TaskFlow API, dynamic DAG generation from config, branching logic, and sensor-based external dependencies. - Error Handling & Alerts: Failure callbacks, trigger rules, retry strategies with exponential backoff, and cleanup tasks that run regardless of upstream failures. - Testing Framework: Pytest-based DAG integrity tests covering import errors, structure validation, dependency checks, and cycle detection. - Use Case: You need to orchestrate a daily ETL pipeline that waits for files on S3, branches based on data quality scores, and alerts your team on failures. Use this Skill to generate the complete DAG with sensors, BranchPythonOperator, and failure callbacks. ## Quick Start Create an Airflow DAG for a daily ETL pipeline that extracts data from S3, transforms it, loads it to a data warehouse, and sends a Slack alert on failure.
