senior-data-engineer

Design and implement scalable data pipelines using Python, SQL, Spark, Airflow, dbt, and Kafka.

Updated Mar 12, 2026

Search Tags:#data pipeline #streaming #data modeling #data engineering #etl #dataops #data warehousing

Ready to run

npx skills add https://github.com/Fantasia1999/claude-skills-zh --skill senior-data-engineer-fantasia1999

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: senior-data-engineer
Source: https://github.com/Fantasia1999/claude-skills-zh/tree/main/translations/engineering-team/senior-data-engineer
Command: npx skills add https://github.com/Fantasia1999/claude-skills-zh --skill senior-data-engineer-fantasia1999

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the complexities of building and maintaining scalable, reliable data systems, from ETL pipelines to robust data infrastructure.

Core Features & Use Cases

Pipeline Design & Orchestration: Design and implement batch and streaming data pipelines using tools like Airflow, Spark, and Kafka.
Data Modeling & Warehousing: Create efficient data models (dimensional, Data Vault) and leverage modern data warehouses and lakehouses.
Data Quality & Governance: Implement data validation, monitoring, and DataOps best practices.
Use Case: A company needs to ingest customer transaction data from multiple sources, transform it, and load it into a data warehouse for business intelligence. This Skill can guide the design and implementation of such a pipeline, ensuring scalability and data quality.

Quick Start

Use the senior-data-engineer skill to design a batch ETL pipeline from a PostgreSQL source to a Snowflake destination, scheduled daily at 5 AM.

Frequently Asked Questions about senior-data-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I design a scalable ETL pipeline from PostgreSQL to Snowflake?▼

To design a scalable ETL pipeline from PostgreSQL to Snowflake, you orchestrate data ingestion and transformation using Python, Airflow, and dbt. This ensures automated batch processing, data quality validation, and reliable warehouse loading.

What is the best way to orchestrate batch and streaming data pipelines?▼

The best way to orchestrate batch and streaming data pipelines is using Airflow for scheduling, Spark for distributed processing, and Kafka for streaming ingestion. This combination ensures robust data infrastructure and scalable DataOps workflows.

How does data modeling work in a modern data warehouse?▼

Data modeling in a modern data warehouse involves structuring data using dimensional models or Data Vault architectures. This approach optimizes analytical querying and supports scalable data infrastructure within lakehouse environments.

Can I use dbt and Kafka for real-time data ingestion and transformation?▼

Yes, you can use Kafka for real-time data ingestion and dbt for SQL-based transformations within your data pipeline. This combination supports both streaming and batch processing workflows while maintaining data quality and DataOps principles.

Do I need Python and SQL expertise to implement DataOps best practices?▼

Yes, you need Python and SQL expertise to implement DataOps best practices, as they are fundamental for writing data transformations, orchestrating pipelines with Airflow, and ensuring data quality validation across your infrastructure.

When should I choose Data Vault modeling over dimensional modeling for my data warehouse?▼

You should choose Data Vault modeling over dimensional modeling when your data warehouse requires high scalability for integrating multiple sources and historical tracking, whereas dimensional modeling is better suited for optimized business intelligence querying.