senior-data-engineer

Design scalable data pipelines by defining data models and orchestrating ETL/ELT workflows with Airflow. This approach supports both batch and streaming workloads across modern data stacks using Python, SQL, Spark, and dbt.

DataOps applies robust error handling, data quality checks, and observability to data pipeline orchestration. It integrates with Airflow and dbt to ensure reliable ETL/ELT workflows and data governance across your stack.

Yes, you can use dbt and Airflow together for ETL workflows. Airflow handles pipeline orchestration and scheduling, while dbt manages SQL transformations and data modeling within your modern data stack.

Set up a minimal end-to-end data pipeline by initializing Python scripts and Spark processing jobs orchestrated by Airflow. This demonstrates core pipeline functionality, monitoring, and orchestration capabilities.

Implement data governance and data quality by applying DataOps principles with Airflow and dbt. This establishes robust error handling, observability, and governance frameworks across your scalable data pipelines.