skills.rest
bcastelino
bcastelinoCOMMUNITY
Explore all skills by this author →
Website𝕏@cas7elinoGitHub

data-engineering-data-pipeline

Design and implement batch and streaming data pipelines with Airflow, dbt, and Delta Lake.

2|Updated Feb 6, 2026
Search Tags:#data pipeline#streaming#dbt#data engineering#etl#batch processing#elt
One-click install
npx skills add https://github.com/bcastelino/agent-skills-kit --skill data-engineering-data-pipeline
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-engineering-data-pipeline
Source: https://github.com/bcastelino/agent-skills-kit/tree/main/skills/data-engineering-data-pipeline
Command: npx skills add https://github.com/bcastelino/agent-skills-kit --skill data-engineering-data-pipeline

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill provides comprehensive guidance and best practices for designing, implementing, and optimizing scalable, reliable, and cost-effective data pipelines for both batch and streaming data.

Core Features & Use Cases

  • Architecture Design: Choose and design ETL/ELT, Lambda, Kappa, or Lakehouse architectures.
  • Implementation: Build ingestion, transformation (dbt, Spark), orchestration (Airflow, Prefect), and storage (Delta Lake, Iceberg) layers.
  • Data Quality & Monitoring: Implement data quality frameworks and set up robust monitoring and cost optimization strategies.
  • Use Case: Design a streaming data pipeline to ingest real-time user activity, process it, and store it in a Delta Lake for immediate analytics.

Quick Start

Design a batch data pipeline for processing daily sales orders, including ingestion, transformation with dbt, and storage in Delta Lake.

Frequently Asked Questions about data-engineering-data-pipeline

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I design a data pipeline for both batch and streaming data?▼

Design scalable data pipelines by selecting architectures like Lambda, Kappa, or Lakehouse to handle both batch and streaming data, ensuring reliable ingestion, transformation, and storage.

What's the best way to orchestrate ETL workflows with Airflow and dbt?▼

Orchestrate ETL workflows by integrating Airflow for scheduling and dbt for transformations, creating reliable data pipelines that process batch data and load it into storage layers like Delta Lake.

When should I use ELT instead of ETL for data transformation?▼

Choose ELT over ETL when loading raw data directly into a data lake or warehouse first, then using tools like Spark or dbt for transformation, optimizing for scalability and cost-effectiveness.

How do I implement data quality frameworks in a streaming data pipeline?▼

Implement data quality frameworks within streaming pipelines by validating data during ingestion and transformation, monitoring pipeline health to ensure reliable real-time analytics and storage.

Does this approach support building a Lakehouse architecture with Delta Lake and Iceberg?▼

Yes, you can build a Lakehouse architecture by utilizing Delta Lake and Iceberg for storage management, combining the reliability of data warehouses with the flexibility of data lakes.

How do I monitor pipeline health and optimize data processing costs?▼

Monitor pipeline health by tracking operational metrics and applying cost optimization strategies during ingestion and transformation, ensuring your batch and streaming data pipelines remain reliable and efficient.

Related Skills

Explore More Software Engineering Skills→
zangxin75zangxin75

gcloud

Manage Google Cloud Platform resources via gcloud CLI, gsutil, and Firebase tools.

Community
Advanced
weirdgmeweirdgme
4

platform-engineering

Design and implement scalable platform engineering patterns for internal developer platforms.

Community
Advanced
FrekiManagarmFrekiManagarm

source-command-sparc-devops

Automate DevOps workflows for deployment, provisioning, configuration, monitoring, and cleanup.

Community
Advanced
AdawoduAdawodu

engineering-data-engineer

Design and implement ETL/ELT pipelines with Apache Spark and dbt.

Community
Advanced