senior-data-engineer

To design scalable data pipelines for batch and streaming contexts, architect modular data models and ELT workflows using Airflow, Spark, Kafka, and dbt to ensure repeatable, auditable data flows across cloud data lakes and warehouses.

Establishing data quality and governance across cloud data warehouses requires implementing data contracts, lineage tracking, and quality checks using Great Expectations alongside modern data stack practices to protect organizational data trust.

Yes, you can orchestrate end-to-end ETL workflows by integrating dbt for data transformations with Airflow for scheduling and automation, ensuring repeatable and auditable data flows across your infrastructure.

To set up a baseline data pipeline with governance scaffolding, initialize the starter project to establish foundational ETL workflows, data models, and quality checks for reliable analytics.

Yes, architecting robust data infrastructure requires deep expertise in Python and SQL, alongside Spark, Airflow, dbt, and Kafka, to build scalable architectures supporting ETL/ELT workflows and DataOps.

Use Kafka for real-time data pipelines to handle streaming data ingestion and event routing, while Spark processes large-scale batch transformations, enabling combined batch and streaming analytics platforms.