skills.rest
duggal1
duggal1COMMUNITY
Explore all skills by this author →
Website𝕏@harshitduggal5GitHub

dag-orchestration-patterns

Document reusable DAG patterns and retry strategies for ETL/ELT workflows.

2|1|Updated Mar 8, 2026
Search Tags:#orchestration#etl#airflow#data-pipelines#prefect#dagster#dag-patterns
One-click install
npx skills add https://github.com/duggal1/Sapphire-cli --skill dag-orchestration-patterns
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: dag-orchestration-patterns
Source: https://github.com/duggal1/Sapphire-cli/tree/main/skills/data-pipeline/.claude/skills/dag-orchestration-patterns
Command: npx skills add https://github.com/duggal1/Sapphire-cli --skill dag-orchestration-patterns

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Data engineers need reusable, proven DAG patterns to reliably orchestrate ETL/ELT pipelines across tools like Airflow, Dagster, and Prefect.

Core Features & Use Cases

  • ELT pattern: structure for extracting data, loading raw, and transforming for analytics.
  • Fan-out / Fan-in: parallelize source data processing and merge results for scalable pipelines.
  • Event-pending and MERGE/UPSERT patterns: coordinate data readiness and up-to-date targets with robust retry and idempotency.
  • Partition management and common orchestration strategies for reliable scheduling and monitoring.

Quick Start

Select an orchestration pattern (ELT, fan-out, or event pending) and implement it in your DAG using your preferred toolchain.

Frequently Asked Questions about dag-orchestration-patterns

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
What are the best DAG patterns for orchestrating ETL pipelines in Airflow?▼

Reusable DAG patterns for ETL orchestration include ELT structures, fan-out/fan-in parallelization, and event-pending workflows. These patterns provide standardized structures for extracting, loading, and transforming data reliably across orchestrators like Airflow.

How do I handle idempotency and retries when managing data pipelines?▼

To handle idempotency and retries in data pipelines, implement MERGE/UPSERT patterns alongside robust retry strategies. This ensures data readiness and up-to-date targets by safely reprocessing failed batches without duplicating records.

Can I use these orchestration patterns for both batch and streaming data scenarios?▼

Yes, these orchestration patterns apply to both batch and streaming data scenarios. They provide recommended configurations and common strategies for reliable scheduling, monitoring, and partition management across diverse ETL/ELT workflows.

How do I parallelize source data processing to scale my ELT workflows?▼

To scale ELT workflows, implement fan-out/fan-in patterns to parallelize source data processing. This approach distributes extraction tasks across multiple workers and merges the results back together for efficient analytics.

Does this provide examples for integrating and testing DAGs in Dagster and Prefect?▼

Yes, it provides concrete examples and guidance for integrating and testing DAGs. These configurations apply to teams implementing ETL/ELT workflows across popular orchestrators including Dagster and Prefect.

Related Skills

Explore More Data & Analytics Skills→
muzhicaomingwangmuzhicaomingwang
3

data-warehouse

Consolidate data sources into a single source of truth for analytics.

Community
Advanced
ivanshamaevivanshamaev
14

de-production-readiness

Configure idempotent writes, retries, SLA alerts, and observability for data pipelines.

Community
Advanced
Harvest-Forged-CodeHarvest-Forged-Code
10

data-pipeline-design

Guide ETL/ELT and streaming pipeline design with error handling and monitoring.

Official
Advanced
AI-Foundry-CoreAI-Foundry-Core
4

airflow-dag-patterns

Design and implement production Apache Airflow DAGs with standardized patterns.

Official
Advanced