skills.rest
williamzujkowski
williamzujkowskiCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

Data Engineering Pipeline Designer

Design data pipelines with quality checks, orchestration, and governance.

6|Updated Oct 25, 2025
Search Tags:#orchestration#dbt#airflow#kafka#data-pipelines#data-engineering#great-expectations
One-click install
npx skills add https://github.com/williamzujkowski/cognitive-toolworks --skill data-engineering-pipeline-designer
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: Data Engineering Pipeline Designer
Source: https://github.com/williamzujkowski/cognitive-toolworks/tree/main/skills/data-pipeline-designer
Command: npx skills add https://github.com/williamzujkowski/cognitive-toolworks --skill data-engineering-pipeline-designer

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Streamlines the design and governance of complex data pipelines by providing a repeatable blueprint that captures requirements, architecture, and deployment steps.

Core Features & Use Cases

  • End-to-end pipeline design with quality checks, orchestration, and governance patterns.
  • Supports batch, streaming, and hybrid data workflows using modern stacks (Airflow, dbt, Kafka, Great Expectations).
  • Use cases include building reliable ELT pipelines for data warehouses and lakehouses, with lineage and monitoring baked in.

Quick Start

Provide a complete, ready-to-implement data-pipeline design by specifying inputs like pipeline_type, sources, transformations, quality checks, and orchestration platform, then generate deployment steps.

Frequently Asked Questions about Data Engineering Pipeline Designer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I design a data pipeline with dbt and Airflow orchestration?▼

To design data pipelines with dbt and Airflow, specify your pipeline type, sources, transformations, and orchestration platform to generate ready-to-implement DAG templates, dbt patterns, and deployment steps.

What's the best way to add data quality checks to an ELT pipeline?▼

Adding data quality checks to an ELT pipeline involves integrating Great Expectations configurations alongside your dbt transformations, ensuring automated validation and governance controls are baked into the pipeline architecture.

Can I build streaming data pipelines with Kafka using this approach?▼

Yes, you can build streaming data pipelines with Kafka by specifying a streaming pipeline type, enabling real-time data workflows alongside batch processing to support hybrid architectures across lakes and warehouses.

How do I set up governance and lineage for a data warehouse pipeline?▼

Setting up governance and lineage for a data warehouse pipeline requires applying governance controls and monitoring patterns during the design phase, ensuring data tracking and compliance are integrated into the core workflow.

Does this pipeline design approach support both batch and hybrid workflows?▼

Yes, this pipeline design approach supports batch, streaming, and hybrid data workflows, allowing you to generate comprehensive blueprints that capture architecture and deployment steps for analytics and BI.

Related Skills

Explore More Data & Analytics Skills→
hsigstadhsigstad

new-pipeline

Generate a new data pipeline directory structure with boilerplate documentation.

Community
Intermediate
borgheiborghei
467

senior-ml-engineer

Provide production-grade ML engineering guidance and templates for end-to-end projects.

Community
Advanced
htlin222htlin222
77

data-engineer

Designs and implements scalable data pipelines, warehouses, and streaming architectures.

Community
Advanced
mdnaimul22mdnaimul22
2

senior-data-engineer

Design production-grade data pipeline architectures for batch and streaming ETL/ELT systems.

Community
Advanced