senior-data-engineer

This Skill generates Airflow DAGs to orchestrate ETL pipelines, extracting data from PostgreSQL, transforming it with dbt, and loading it into Snowflake. It includes scheduling, monitoring, and data quality checks for reliable pipeline execution.

Designing scalable data pipelines involves architecting batch ETL/ELT processes, real-time streaming flows with Kafka, and data lakehouse solutions. This Skill provides blueprints for pipeline orchestration, data modeling, and DataOps to ensure infrastructure reliability and scalability.

Implementing data quality and governance in dbt workflows requires establishing data contracts for reliable exchange and designing efficient data models. This Skill guides setting up robust monitoring, data quality validation, and governance frameworks within your data architecture.

Yes, you can use Spark and Python for data lakehouse ETL orchestration. This Skill supports building data infrastructure using Python, SQL, and Spark, integrating them with Airflow for automated workflow scheduling, pipeline optimization, and reliable DataOps execution.

Use Kafka for real-time streaming when your data infrastructure requires low-latency data ingestion and continuous processing, rather than scheduled batch ETL pipelines. This Skill helps architect both streaming and batch solutions to fit your specific data architecture needs.