senior-data-engineer

Design scalable batch and streaming data pipelines by providing a dataset and objective to generate a production-grade data engineering plan. It orchestrates workflows with Airflow, dbt, and Spark Streaming.

Implement real-time data processing with Kafka by applying this skill's templates for streaming ingestion and pipeline orchestration. It supports Flink and Spark Streaming for production-grade real-time analytics.

Add data quality gates to an ETL pipeline by integrating DataOps, governance, and quality tooling included in the skill. It provides best practices for ensuring reliable analytics at scale across cloud data stacks.

You can use Python and SQL for end-to-end pipeline orchestration. The skill supports modern data tooling including Airflow and dbt to automate both batch ETL and streaming ingestion workflows.

This skill supports DataOps for production-grade data platforms by automating end-to-end workflows. It includes tooling and best practices for data quality, governance, and reliable analytics at scale.

Separate batch ETL and streaming ingestion workflows when building analytics platforms requiring both historical processing and real-time analytics. The skill provides templates to manage both workload types concurrently.