skills.rest
RDEWAI
RDEWAIOFFICIAL
Explore all skills by this author →
GitHub

create-dag

Generate Apache Airflow DAG files from LLD artifacts and pipeline configs.

5|1|Updated Sep 23, 2025
Search Tags:#data engineering#airflow#spark#medallion architecture#low-level design#dag generation#pipeline scaffolding
One-click install
npx skills add https://github.com/RDEWAI/Redefining-DataEngineering-With-AI --skill create-dag
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: create-dag
Source: https://github.com/RDEWAI/Redefining-DataEngineering-With-AI/tree/main/chapter-6/developer-plugin/skills/create-dag
Command: npx skills add https://github.com/RDEWAI/Redefining-DataEngineering-With-AI --skill create-dag

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Manually translating low-level design (LLD) artifacts into Apache Airflow DAGs is time-consuming, error-prone, and often leads to inconsistent pipeline configurations that fail validation or require extensive rework.

Core Features & Use Cases

  • Automated DAG Generation: Translates approved LLD markdown and pipeline configs into production-ready Airflow DAG files following project coding patterns and naming conventions.
  • Automatic Spark Wrapper Creation: Generates per-task entry shims for SparkSubmitOperator to avoid runtime permission errors and ensure consistent task execution.
  • Built-in Validation & Compliance: Runs DAG path validation, checks library cache freshness, and verifies acceptance criteria compliance before marking work complete.
  • Use Case: Data engineering teams building medallion architecture (bronze/silver/gold) pipelines can use this skill to eliminate manual scaffolding, reduce configuration errors, and speed up deployment of new data workflows.

Quick Start

Use the create-dag skill to generate a production-ready Airflow DAG from the approved low-level design for your patient_360 medallion pipeline.

Frequently Asked Questions about create-dag

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I generate Apache Airflow DAGs from low-level design documents?▼

To generate Apache Airflow DAGs from low-level designs, the Skill translates approved LLD markdown and pipeline configuration files into production-ready DAG files. It automatically applies project coding patterns and naming conventions to eliminate manual scaffolding.

How do I scaffold medallion architecture pipelines in Airflow?▼

Scaffolding medallion architecture pipelines in Airflow involves generating bronze, silver, and gold data workflow DAG files. The Skill automates this pipeline scaffolding from your approved LLD artifacts to ensure consistent data engineering configurations.

Do I need to manually create Spark task wrappers for my Airflow DAG?▼

You do not need to manually create Spark task wrappers for your Airflow DAG. The Skill generates per-task entry shims for the SparkSubmitOperator automatically, preventing runtime permission errors and ensuring consistent task execution.

Can I validate Airflow 3.x DAG path rules and library cache before deployment?▼

You can validate Airflow 3.x DAG path rules and library cache before deployment. The Skill runs built-in DAG path validation, checks library cache freshness, and verifies acceptance criteria compliance against Airflow 3.x versioning standards.

What is the best way to translate pipeline configurations into Airflow DAG files?▼

The best way to translate pipeline configurations into Airflow DAG files is using an automated generation approach that reads approved configs and LLDs. This method ensures compliance with path rules and eliminates inconsistent manual coding errors.

Why do manually created Airflow DAGs fail validation against path rules?▼

Manually created Airflow DAGs fail validation against path rules due to inconsistent pipeline configurations and human error during scaffolding. Automating DAG generation from approved LLDs ensures compliance with project coding patterns and validation requirements.

Related Skills

Explore More Data & Analytics Skills→
linhlvitlinhlvit

datamart-gen-docs

Generate PTTK and TKCSLD design documents as Markdown and DOCX.

Community
Advanced
rory-datarory-data

testing-dags

Trigger Airflow DAG runs, diagnose failures, inspect logs, and apply fixes in iterative cycles.

Community
Advanced
pypeadaypypeaday
2

airflow

Build and debug Airflow DAGs with KubernetesPodOperator across versions 2.x and 3.x.

Community
Advanced
RDEWAIRDEWAI
5

validate-dag

Validate Airflow DAG files for import errors and coding standards.

Official
Intermediate