senior-data-engineer

Design and operate ETL/ELT pipelines across cloud data stacks.

1|1|Updated Mar 22, 2026
One-click install
npx skills add https://github.com/zzafergok/skills --skill senior-data-engineer-zzafergok
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: senior-data-engineer
Source: https://github.com/zzafergok/skills/tree/main/10-research-data/senior-data-engineer
Command: npx skills add https://github.com/zzafergok/skills --skill senior-data-engineer-zzafergok

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

Building and maintaining scalable data pipelines and robust data infrastructure for production analytics and ML workloads.

Core Features & Use Cases

  • Data modeling, pipeline orchestration, and governance across batch and real-time data processing.
  • ETL/ELT design, quality checks, and DataOps practices to ensure reliability and observability.
  • Use Case: Design an end-to-end data stack for a fintech analytics platform with streaming ingest, validation, and model-ready features.

Quick Start

Create a production-grade data pipeline for a new data product and validate data quality end-to-end.

Frequently Asked Questions about senior-data-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build scalable data pipelines for production analytics and ML workloads?

Build scalable data pipelines by applying DataOps practices, ETL/ELT design, and pipeline orchestration using Python, SQL, Spark, and Airflow to ensure reliability and observability for production analytics and ML workloads.

What is the best way to design an end-to-end data stack with streaming ingest and validation?

Design an end-to-end data stack using Kafka for streaming ingest, dbt for data modeling, and automated quality checks to validate data and deliver model-ready features for analytics platforms.

How do I orchestrate ETL systems and maintain data quality across cloud data stacks?

Orchestrate ETL systems and maintain data quality across cloud data stacks by leveraging Airflow for pipeline orchestration, dbt for transformations, and DataOps practices for governance and observability.

Can I use dbt and Spark together for batch and real-time data processing?

Yes, you can use dbt for data modeling and transformations alongside Spark for processing batch and real-time data, ensuring robust data infrastructure and pipeline scalability across your stack.

What DataOps practices should I apply to ensure data pipeline security and governance?

Apply DataOps practices like automated quality checks, pipeline observability, and validation protocols to ensure security, governance, and reliability across your data infrastructure and ETL systems.