data-pipeline-engineer

A medallion architecture organizes ETL pipelines into Bronze (raw), Silver (cleaned), and Gold (business-ready) layers. Use dbt for transformations and Spark for distributed processing, defining models at each layer with incremental loads and data quality tests to ensure idempotent, reproducible transformations across batch and streaming data.

Yes. Airflow schedules and monitors dbt workflows as DAGs, triggering dbt models on a schedule or event. This combination handles orchestration logic, retry policies, alerting, and backfills while dbt manages transformation logic, testing, and data lineage for end-to-end pipeline automation.

Kafka ingests event streams; Spark Structured Streaming processes them with watermarks and stateful operations. Implement SCD (Slowly Changing Dimensions) patterns, idempotent writes to Delta Lake, and quality gates to transform streaming events reliably into warehouse tables.

Embed data quality gates using dbt tests, Great Expectations, or custom validations at each medallion layer. Test for schema compliance, referential integrity, and business rules before data moves downstream, ensuring only validated data reaches downstream consumers and enabling automated alerting on failures.

Both support incremental loads, partitioning, and ACID transactions for reliable ETL. Delta Lake integrates tightly with Spark and dbt; Iceberg offers format flexibility. Choose based on your cloud platform and ecosystem; both enable idempotent transformations and historical tracking.

You need a data source (batch or streaming), a data warehouse or lakehouse (Delta Lake, Iceberg, or Redshift), Airflow or Dagster for orchestration, dbt for transformation modeling, and Spark for distributed processing. Ensure connectivity, schema design, and monitoring infrastructure before deploying.