senior-data-engineer

Name: senior-data-engineer
Availability: InStock
Author: questnova502

Design and operate production-grade ETL/ELT data pipelines with Python, Spark, and Airflow.

4|5|Updated Jan 19, 2026

Search Tags:#data #pipeline #engineering #etl #airflow #spark #dataplatform

Ready to run

npx skills add https://github.com/QuestNova502/claude-skills-sync --skill senior-data-engineer-questnova502

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: senior-data-engineer
Source: https://github.com/QuestNova502/claude-skills-sync/tree/main/skills/senior-data-engineer
Command: npx skills add https://github.com/QuestNova502/claude-skills-sync --skill senior-data-engineer-questnova502

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

Provides a cohesive, production-grade skill set for designing, building, and governing scalable data pipelines, ETL/ELT workflows, and data infrastructure to support analytics and ML initiatives.

Core Features & Use Cases

End-to-end data pipeline design and implementation (ETL/ELT) with Python, Spark, and orchestration via Airflow.
Data modeling, pipeline orchestration, and DataOps best practices for maintainability and observability.
Use cases include building scalable data platforms, batch and streaming processing, and data governance/compliance workflows.

Quick Start

Set up a production-grade data pipeline using Python, Spark, and Airflow to orchestrate ETL/ELT workloads.

Frequently Asked Questions about senior-data-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I design a scalable data pipeline for ETL and batch processing?▼

Build scalable data pipelines by orchestrating ETL/ELT workflows with Python, Spark, and Airflow DAGs to achieve production-grade, fault-tolerant batch processing and streaming ingestion across modern data architectures.

What is the best way to orchestrate Airflow DAGs for streaming ingestion and data quality checks?▼

Orchestrate Airflow DAGs to automate streaming ingestion, batch processing, and data quality checks, applying DataOps best practices for pipeline observability, maintainability, and automated data governance.

How do I apply data modeling and governance in a modern data platform?▼

Apply data modeling and governance workflows within your data platform by integrating DataOps best practices, ensuring data quality checks, security, and compliance across analytics and ML pipelines.

Can I use Spark and Airflow for real-time processing and ML data pipelines?▼

Yes, you can use Spark and Airflow to build real-time processing and ML pipelines, supporting scalable architectures that handle streaming ingestion, fault tolerance, monitoring, and cost optimization.

What are the limitations of batch processing in ETL workflows?▼

Batch processing in ETL workflows can introduce latency compared to streaming ingestion; however, combining it with Airflow orchestration and data quality checks ensures fault tolerance and scalable architectures.