data-engineering-data-pipeline

What problem does it solve?

This Skill provides comprehensive guidance and best practices for designing, implementing, and optimizing scalable, reliable, and cost-effective data pipelines for both batch and streaming data.

Core Features & Use Cases

• Architecture Design: Choose and design ETL/ELT, Lambda, Kappa, or Lakehouse architectures.
• Implementation: Build ingestion, transformation (dbt, Spark), orchestration (Airflow, Prefect), and storage (Delta Lake, Iceberg) layers.
• Data Quality & Monitoring: Implement data quality frameworks and set up robust monitoring and cost optimization strategies.
• Use Case: Design a streaming data pipeline to ingest real-time user activity, process it, and store it in a Delta Lake for immediate analytics.

Quick Start

Design a batch data pipeline for processing daily sales orders, including ingestion, transformation with dbt, and storage in Delta Lake.