agency-data-engineer

Idempotent ETL pipelines for a lakehouse architecture are built by applying Bronze→Silver→Gold medallion patterns with clear data contracts. This approach designs batch and streaming workflows that safely reprocess raw, messy data into reliable, analytics-ready datasets without duplication.

Handling schema drift in streaming ingestion from Kafka requires schema validation, data contracts, and Delta/Iceberg table strategies. These mechanisms ensure raw data is cleanly ingested, deduplicated, and transformed into analytics-ready datasets for BI and ML consumers.

Data quality validation and lineage in dbt models are established through dbt contracts, Great Expectations checks, and SLA monitoring. This combination eliminates silent data failures by ensuring schema validation and observability across the entire data pipeline.

Spark Structured Streaming supports incremental CDC pipelines by minimizing cost and latency during cloud-native ingestion. It processes Change Data Capture events from sources like Kafka or Event Hubs into Delta or Iceberg tables for downstream analytics.

Partitioning and compaction strategies for Delta Lake tables are needed to optimize storage and query performance in lakehouse architectures. Applying these techniques alongside incremental processing prevents high-cost full refreshes and ensures timely data delivery.

This approach works with Databricks, Snowflake, BigQuery, and Fabric by providing platform-specific patterns for Delta, Iceberg, and Hudi table formats. It ensures idempotent ETL/ELT pipelines and medallion architectures operate reliably across different cloud-native environments.