senior-data-engineer

Plans and designs scalable data pipelines and architectures using Python, SQL, Spark, Airflow, dbt, and Kafka.

2|Updated Feb 17, 2026
One-click install
npx skills add https://github.com/Haseeb-Arshad/TaskHive --skill senior-data-engineer-haseeb-arshad
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: senior-data-engineer
Source: https://github.com/Haseeb-Arshad/TaskHive/tree/main/.claude/skills/senior-data-engineer
Command: npx skills add https://github.com/Haseeb-Arshad/TaskHive --skill senior-data-engineer-haseeb-arshad

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Data teams struggle to design, implement, and operate scalable data pipelines and governance structures across rapidly growing data volumes and diverse tech stacks.

Core Features & Use Cases

  • End-to-end data architecture design, pipeline orchestration, data quality, and DataOps guidance for Python, SQL, Spark, Airflow, dbt, and Kafka.
  • Use cases include building scalable data platforms, implementing robust ETL/ELT processes, and governing data through modeling, lineage, and quality checks.

Quick Start

Design an end-to-end data architecture for a new analytics platform and outline the first implementation steps.

Frequently Asked Questions about senior-data-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I architect scalable data pipelines for enterprise-grade analytics?

To architect scalable data pipelines, you plan end-to-end data architectures using Python, SQL, Spark, and Airflow, applying data modeling, orchestration, and quality checks to ensure robust ingestion and governance across large-scale data platforms.

What is the best way to orchestrate ELT processes with Airflow and dbt?

The best way to orchestrate ELT processes with Airflow and dbt is to design DataOps workflows that integrate pipeline orchestration, data quality guardrails, and lineage tracking, ensuring production-grade data transformations and reliable execution.

Can I use this approach to design data governance and quality checks for Kafka streams?

Yes, you can design data governance and quality checks for Kafka streams by applying enterprise-grade data architecture frameworks that define modeling, lineage, and quality guardrails across diverse ingestion scenarios.

How do I implement DataOps guardrails and decision frameworks for Spark data engineering?

You implement DataOps guardrails for Spark data engineering by establishing defined decision frameworks that enforce production-grade requirements, including data quality checks, orchestration rules, and governance policies throughout the pipeline lifecycle.

When do I need to plan enterprise data architecture for ingestion and governance?

You need to plan enterprise data architecture when struggling to operate scalable data pipelines across rapidly growing data volumes, requiring structured DataOps, data modeling, and orchestration to manage diverse tech stacks effectively.