skills.rest
hiddink-ai
hiddink-aiOFFICIAL
Explore all skills by this author →
GitHub

pipeline-architecture-patterns

Guide data pipeline architecture with ETL/ELT, orchestration, and data quality patterns.

1|Updated May 21, 2026
Search Tags:#architecture#data quality#orchestration#data engineering#etl#elt
One-click install
npx skills add https://github.com/hiddink-ai/hiddink-harness --skill pipeline-architecture-patterns-hiddink-ai
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: pipeline-architecture-patterns
Source: https://github.com/hiddink-ai/hiddink-harness/tree/main/templates/skills/pipeline-architecture-patterns
Command: npx skills add https://github.com/hiddink-ai/hiddink-harness --skill pipeline-architecture-patterns-hiddink-ai

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This skill addresses the complexity of designing and maintaining data pipelines by providing standardized architectural patterns for ETL/ELT, orchestration, and data quality.

Core Features & Use Cases

  • Architectural Blueprints: Compare and implement ETL vs ELT, Lambda, Kappa, and Medallion architectures based on your specific data latency and complexity needs.
  • Orchestration Strategies: Learn to implement DAG-based, event-driven, or hybrid orchestration to manage task dependencies and scheduling.
  • Data Quality & Integrity: Utilize data contracts, validation frameworks like Great Expectations, and idempotency patterns to ensure reliable, production-grade data delivery.

Quick Start

Ask the agent to evaluate your current data pipeline architecture and suggest improvements based on the pipeline-architecture-patterns skill.

Frequently Asked Questions about pipeline-architecture-patterns

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I design robust ETL and ELT data pipelines for production environments?▼

To design robust ETL and ELT data pipelines, utilize architectural blueprints like Lambda, Kappa, and Medallion architectures to match your specific data latency and complexity needs. This ensures scalable and reliable processing in production.

What is the best way to implement data quality frameworks and validation in data engineering?▼

Implementing data quality frameworks requires utilizing data contracts and validation frameworks like Great Expectations. This approach enforces schema rules and ensures reliable, production-grade data delivery across your pipelines.

How do I choose between DAG-based and event-driven orchestration for data pipelines?▼

Choosing between DAG-based and event-driven orchestration depends on your task dependencies and scheduling needs. Compare these hybrid orchestration strategies to manage complex pipeline workflows and optimize data processing efficiency.

Does this architectural guidance support idempotent design and data lineage tracking?▼

Yes, this architectural guidance satisfies requirements for idempotent design, data lineage tracking, and schema enforcement. These patterns ensure your data pipelines maintain integrity and reliability in production environments.

Related Skills

Explore More Data & Analytics Skills→
lrbjklrbjk

unity-project-scout

Assess Unity projects for version, render pipeline, packages, asmdefs, and folder structure.

Community
Intermediate
gaiastudio-aigaiastudio-ai
2

gaia-post-deploy

Run post-deploy health checks, smoke tests, and metrics validation against SLOs.

Official
Advanced
vignesh2027vignesh2027
6

data-governance-agent

Design enterprise data governance frameworks with cataloging, lineage, and PII policies.

Community
Advanced
vertivolatamvertivolatam
1

Google Cloud Platform Backend

Deploy and manage Flutter application backends on Google Cloud Platform.

Official
Advanced