senior-data-engineer

Design scalable data pipelines by applying architecture design, ETL/ELT orchestration, and data modeling across Python, SQL, Spark, Airflow, dbt, and Kafka to achieve production-grade reliability and scalable data operations.

Orchestrate ETL workflows by integrating Airflow for pipeline scheduling and dbt for data modeling and transformations, ensuring end-to-end execution, observability, and DataOps governance across analytics platforms.

Implement data quality and governance in Spark pipelines through validation rules, observability monitoring, and DataOps governance frameworks to maintain production-grade reliability and accurate analytics outputs.

Build real-time streaming pipelines using Kafka and Python to ingest data into scalable data lakes, enabling enterprise data platforms with production-grade reliability and continuous data operations.

Use dbt for data modeling when orchestrating scalable ELT workflows with built-in governance and data quality validation, whereas raw SQL suits isolated transformations lacking pipeline observability and orchestration requirements.

Data pipeline orchestration fails to scale without proper architecture design, distributed processing via Spark, and robust DataOps governance, leading to bottlenecks in ETL/ELT workflows and degraded observability.