senior-data-engineer

Name: senior-data-engineer
Availability: InStock
Author: yezannnnn

Design scalable data pipelines and governance platforms for modern data stacks.

148|50|Updated Feb 12, 2026

Search Tags:#data #dbt #etl #airflow #pipelines #spark

Ready to run

npx skills add https://github.com/yezannnnn/agentGroup --skill senior-data-engineer-yezannnnn

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: senior-data-engineer
Source: https://github.com/yezannnnn/agentGroup/tree/main/jarvis/skills/engineering-team/senior-data-engineer
Command: npx skills add https://github.com/yezannnnn/agentGroup --skill senior-data-engineer-yezannnnn

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Data engineering teams struggle to design, implement, and maintain scalable data pipelines and infrastructure across diverse stacks.

Core Features & Use Cases

Data modeling, pipeline orchestration, data quality, and DataOps capabilities for modern data stacks.
Use Case: Designing end-to-end data architectures and governance frameworks for streaming and batch workloads using Python, SQL, Spark, Airflow, and dbt.

Quick Start

Install and initialize a minimal end-to-end data pipeline using Python, Spark, and Airflow to demonstrate orchestration and monitoring.

Frequently Asked Questions about senior-data-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I design scalable data pipelines for batch and streaming workloads?▼

Design scalable data pipelines by defining data models and orchestrating ETL/ELT workflows with Airflow. This approach supports both batch and streaming workloads across modern data stacks using Python, SQL, Spark, and dbt.

What is DataOps and how does it apply to data pipeline orchestration?▼

DataOps applies robust error handling, data quality checks, and observability to data pipeline orchestration. It integrates with Airflow and dbt to ensure reliable ETL/ELT workflows and data governance across your stack.

Can I use dbt and Airflow together for ETL workflows and data modeling?▼

Yes, you can use dbt and Airflow together for ETL workflows. Airflow handles pipeline orchestration and scheduling, while dbt manages SQL transformations and data modeling within your modern data stack.

How do I set up a minimal end-to-end data pipeline using Python and Spark?▼

Set up a minimal end-to-end data pipeline by initializing Python scripts and Spark processing jobs orchestrated by Airflow. This demonstrates core pipeline functionality, monitoring, and orchestration capabilities.

What's the best way to implement data governance and data quality in modern data stacks?▼

Implement data governance and data quality by applying DataOps principles with Airflow and dbt. This establishes robust error handling, observability, and governance frameworks across your scalable data pipelines.