god-data-engineering

What problem does it solve?

This skill helps teams design and implement end-to-end data pipelines that span batch and streaming workloads, enforcing best practices, governance, and robust operations to reduce failures and hallucinations in data processing.

Core Features & Use Cases

• End-to-end pipeline design across Spark, Flink, Airflow, and data platforms (Delta Lake / Iceberg).
• Data quality and governance with schema management, tests, and observability to ensure trust in data products.
• Production-grade workflows including CI/CD, observability instrumentation, and schema evolution strategies for scalable data platforms.

Quick Start

Design and implement a robust batch/streaming data pipeline using Spark/Flink with Airflow orchestration, dbt modeling, and Delta Lake storage, including data quality and governance checks.