Data Engineer

To build a reliable data pipeline with medallion architecture, you design bronze, silver, and gold layers with clear schema contracts, incremental processing, and strict validation. This structure transforms raw data into trusted, analytics-ready assets for downstream use.

Implementing data quality checks in dbt and Spark pipelines requires schema validation, freshness checks, anomaly detection, and lineage tracking. Adding observability and alerting ensures production reliability and prevents silent failures across your data infrastructure.

Yes, you can use Delta Lake and Kafka for streaming CDC ingestion into a lakehouse. This combination supports reliable streaming ingestion, schema enforcement, and incremental processing to maintain governed, monitored tables for analytics.

Handling deduplication and soft deletes in ETL pipelines requires applying schema contracts and validation rules across bronze-silver-gold layers. This approach ensures reliable data processing and maintains clean, governed tables for BI and ML workloads.

You need schema contracts and observability for ELT workflows when transforming messy source data into governed tables. These controls provide lineage tracking and anomaly detection, ensuring production reliability and preventing silent pipeline failures.