senior-data-engineer

Build scalable data pipelines by orchestrating Python and Spark jobs through workflow tools like Airflow, implementing distributed processing for large datasets, and automating ETL workflows to handle fault tolerance and real-time or batch ingestion at production scale.

Implement data quality validation by embedding validation rules and checks directly into your ETL pipeline stages, using frameworks like dbt and Airflow to enforce data accuracy before downstream processing, ensuring reliable data delivery and governance compliance.

Optimize ETL performance by tuning Spark configurations, leveraging Kafka for real-time streaming ingestion, implementing efficient SQL queries, and monitoring pipeline bottlenecks through instrumentation to balance latency and throughput in production environments.

Yes, Airflow and dbt work in synergy for production pipelines: Airflow orchestrates workflow dependencies and scheduling while dbt handles transformation logic, data lineage, and quality testing, enabling modular, maintainable data architecture.

Production patterns include implementing retry logic and idempotency in Airflow tasks, adding monitoring instrumentation for pipeline health, configuring alerts for data quality failures, and designing for cost optimization and security compliance in distributed systems.

Kafka is optional for batch pipelines but essential for real-time streaming scenarios requiring low-latency event ingestion. Batch pipelines can use file-based sources with Spark and Airflow scheduling, while Kafka excels in continuous data ingestion and event-driven architectures.