senior-data-engineer

Design scalable data pipelines using Airflow for orchestration and Spark for distributed processing to manage batch ETL and real-time streaming workflows. This combination provides a production-grade architecture for ingesting, transforming, and loading data into analytics platforms.

Implement DataOps governance and data quality checks in Python by embedding validation rules, monitoring, and automated testing within your pipeline orchestration. This approach ensures reliable, enterprise-grade data infrastructure with consistent quality across ingestion and transformation stages.

You can use dbt and SQL to perform data modeling and ELT transformations directly within your data warehouse. This enables structured analytics engineering, allowing you to define, test, and document data models while maintaining governance across large-scale data platforms.

Build a real-time streaming data architecture with Kafka by configuring topics, producers, and consumers to ingest continuous event streams. This enables low-latency data processing and integrates with Airflow and Spark for end-to-end pipeline orchestration and analytics.

This data engineering framework supports architecture decision frameworks for enterprise data platforms by providing structured patterns for batch ETL, real-time streaming, and governance. It helps evaluate trade-offs across Python, SQL, Spark, and Airflow stacks to design robust infrastructure.

Common troubleshooting patterns for large-scale data pipelines include monitoring job failures in Airflow, debugging Spark memory issues, and validating SQL transformations in dbt. These patterns resolve data quality, orchestration, and performance bottlenecks in production environments.