Develop, debug, and manage Apache Airflow v3 data pipelines.
npx skills add https://github.com/diegoscarabelli/system2 --skill airflow-diegoscarabelli
Please help me install this Agent Skill. Skill: airflow Source: https://github.com/diegoscarabelli/system2/tree/main/src/server/agents/skills/airflow Command: npx skills add https://github.com/diegoscarabelli/system2 --skill airflow-diegoscarabelli
This skill addresses the complexity of managing Apache Airflow v3 pipelines by providing a structured framework for DAG development, debugging, and operational best practices.
Use the airflow skill to analyze the current DAG file and suggest optimizations for task dependencies and execution efficiency.
High-intent search queries and answers about installing and using this skill.
Refactoring legacy Airflow DAGs to the TaskFlow API involves restructuring task dependencies using modern SDK decorators. This transition streamlines pipeline creation, scheduling, and testing while maintaining robust data engineering workflows.
Dynamic task mapping in Airflow allows you to generate parallel tasks at runtime based on input data. This skill guides you through scalable pipeline execution patterns to handle variable workloads efficiently within your DAGs.
Troubleshooting Airflow task failures and scheduling bottlenecks requires analyzing DAG execution logs and dependency configurations. This skill provides expert debugging guidance to resolve production-grade errors and optimize task scheduling.
Resolving XCom limitations in complex data pipelines requires managing data passing between tasks to avoid size constraints. This skill offers structured error handling and debugging strategies to address XCom bottlenecks effectively.
Managing Airflow connections in production environments involves configuring secure credentials for external systems. This skill supports operational management tasks including connection setup and troubleshooting production deployment issues.
Dynamic task mapping should be used instead of standard task dependencies in Airflow when processing variable-length input data that requires parallel execution. This approach ensures scalable and idempotent data engineering workflows.
minyeamerConvert YAML configuration files into executable Apache Airflow DAG definitions.
levnikolaevichAudit observability patterns for logging, health checks, metrics, tracing, and log levels.
chicanoandres702Design and orchestrate scalable batch and streaming data pipelines.
PriyanshKuniyalAutomate Apache Airflow DAG development and deployment for complex workflows.
