airflow-expert

What problem does it solve?

This Skill provides expert-level guidance and code examples for designing, implementing, and managing complex data pipelines using Apache Airflow, addressing challenges in workflow orchestration, task dependency management, and error handling.

Core Features & Use Cases

• DAG Design: Create robust and maintainable Directed Acyclic Graphs (DAGs) using both traditional operators and the TaskFlow API.
• Task Orchestration: Define complex task dependencies, implement branching logic, and manage dynamic task generation.
• Error Handling & Best Practices: Implement retry mechanisms, error callbacks, and follow production-ready patterns for reliable pipeline execution.
• Use Case: Develop a daily ETL pipeline that extracts data from multiple sources, transforms it, loads it into a data warehouse, and includes robust error handling and alerting for failures.

Quick Start

Use the airflow-expert skill to generate a basic Python DAG file for a daily ETL pipeline with extract, transform, and load tasks.