skills.rest
davekim917
davekim917COMMUNITY
Explore all skills by this author →
GitHub

data-engineering

Design robust data pipelines with ETL/ELT, orchestration, and governance patterns.

1|Updated Mar 6, 2026
Search Tags:#orchestration#pipelines#data-quality#data-engineering#pattern-library#data-lakehouse#etl-elt
One-click install
npx skills add https://github.com/davekim917/bootstrap --skill data-engineering-davekim917
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-engineering
Source: https://github.com/davekim917/bootstrap/tree/main/plugins/domain/skills/data-engineering
Command: npx skills add https://github.com/davekim917/bootstrap --skill data-engineering-davekim917

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Data engineering teams often struggle to design, standardize, and govern complex pipelines across batch and streaming workloads. This Skill provides domain-specific patterns and checklists to codify best practices.

Core Features & Use Cases

  • Pattern catalogs for end-to-end pipelines, orchestration, quality, governance, and lakehouse architectures.
  • Practical templates and checklists to review, design, and evolve data platforms (Airflow, Dagster, Prefect, Spark, dbt, Iceberg, Delta Lake, Snowflake, BigQuery).
  • Use cases include building ingestion pipelines, implementing reliable ETL/ELT, and enabling observability and contract-driven data governance.

Quick Start

Describe how to implement a basic data pipeline pattern using this skill.

Frequently Asked Questions about data-engineering

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
What are the best practices for designing robust ETL pipelines with idempotency and schema validation?▼

Robust ETL pipelines require idempotent operations, clear data contracts, schema validation, and partitioning to ensure reliable batch and streaming data processing across lakehouse architectures.

How do I implement data quality and observability checks in Airflow or Dagster orchestration workflows?▼

Implementing data observability in Airflow or Dagster involves applying pattern checklists for quality validation, contract-driven governance, and reliable monitoring across your orchestration workflows.

Can I use this skill for building ingestion pipelines and ETL/ELT processes in Snowflake and BigQuery?▼

Yes, you can apply these data engineering patterns to build ingestion pipelines and implement reliable ETL/ELT processes across platforms including Snowflake, BigQuery, Spark, and dbt.

Does this data engineering pattern library provide guidance on SQL modeling or ML model training?▼

No, this data engineering pattern library explicitly avoids SQL modeling and ML model training guidance, focusing strictly on pipeline orchestration, data quality, governance, and lakehouse architecture.

What is the best way to architect a data lakehouse using Iceberg and Delta Lake?▼

Architecting a data lakehouse with Iceberg or Delta Lake involves applying standardized patterns for partitioning, schema validation, and contract-driven governance to ensure reliable batch and streaming workloads.

Related Skills

Explore More Data & Analytics Skills→
vasilyu1983vasilyu1983
73

data-lake-platform

Automate data lake and lakehouse design and deployment across ingestion, transformation, storage, and orchestration.

Community
Advanced
aws-samplesaws-samples
90

file-processing

Process and analyze CSV, JSON, and text data files with automated cleaning and transformation.

Official
Advanced
ouakarouakar

data-engineer

Automate end-to-end data pipelines from ingestion to analytics-ready datasets.

Community
Advanced
cameronraysmithcameronraysmith
14

preferences-data-modeling

Define data modeling conventions for database schemas and pipelines.

Community
Advanced