senior-data-engineer

Design scalable data pipelines by structuring ingestion, transformation, and orchestration across modern data stacks like Airflow and Spark. This approach ensures robust data infrastructure, operational excellence, and optimized performance for large-scale datasets.

Orchestrate ETL workflows by integrating Airflow for scheduling and Spark for distributed data processing. This combination provides production-grade operational excellence, enabling reliable transformation and monitoring for large-scale data pipelines.

Implement data observability and governance by applying validation, lineage tracking, and security compliance patterns within your ELT workflows. This ensures high data quality and operational monitoring across the entire data infrastructure.

Yes, dbt can be used for transformations within a Spark-based data infrastructure to manage ELT processes. This integration allows teams to apply structured transformation patterns, governance, and data quality checks seamlessly.

Monitor and optimize data pipeline costs by implementing production ops guidance, observability tools, and performance optimization patterns. This reduces infrastructure overhead while maintaining robust data processing and operational excellence.