senior-data-engineer

Scalable data pipelines are designed by defining robust data models and architectures that support both batch and streaming workloads. This approach coordinates ETL workflows using Airflow, dbt, Spark, and Kafka for comprehensive data infrastructure.

The best way to enforce data quality and governance across ETL workflows is by implementing data quality gates, lineage tracking, and operational governance rules. This ensures trust and compliance using validation tools like great_expectations.

Yes, you can use dbt for data modeling within an Airflow orchestrated pipeline. This integration supports pipeline orchestration and automation by coordinating ETL and ELT workflows effectively across your data infrastructure.

Build a near real-time marketing analytics data platform with Kafka and Spark by ingesting customer events, performing stream processing transformations, and serving dashboards. This scalable architecture handles both streaming and batch workloads.

You need YAML and great_expectations to automate data quality checks because they provide configuration management and validation frameworks. These dependencies implement automated data quality gates and lineage tracking within your data pipelines.