senior-data-engineer

To design scalable data pipelines, you must evaluate batch versus streaming requirements and choose between Lambda and Kappa architectures. This approach defines a modern tech stack and produces actionable guidance for pipeline orchestration, data quality, and operability.

Modeling dimensions and facts in a data lakehouse involves applying dimensional modeling techniques like star or snowflake schemas, alongside historical tracking patterns such as SCD and Data Vault. This ensures your data infrastructure remains accurate as complexity grows.

Adding data quality checks to ETL systems involves integrating validation rules and monitoring hooks directly into your pipeline orchestration. This process produces actionable governance guidance and incident-style troubleshooting steps for common failure modes.

You need Data Vault modeling instead of standard star schemas when historical data tracking and auditability are critical for enterprise data warehousing. Data Vault provides robust historical modeling, whereas star schemas optimize query performance for dimensional data.

Yes, this data engineering architecture supports Kafka streaming ingestion and orchestration with Airflow. It applies to both batch and streaming decisions, defining a modern tech stack that integrates pipeline orchestration and event streaming.

ETL ELT pipelines fail data quality validation when DataOps governance and monitoring hooks are improperly configured. Troubleshooting these scenarios requires applying incident-style guidance to identify failure modes and ensure performance and operability.