data-engineer

Build ETL pipelines, data warehouses, and streaming architectures with Spark, Airflow, and Kafka.

1|1|Updated Aug 5, 2025

Search Tags:#streaming #data engineering #etl #airflow #kafka #data warehouse #spark

Ready to run

npx skills add https://github.com/slantview/claude --skill data-engineer-slantview

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: data-engineer
Source: https://github.com/slantview/claude/tree/main/skills/data-engineer
Command: npx skills add https://github.com/slantview/claude --skill data-engineer-slantview

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the complexity of building and maintaining robust data pipelines, data warehouses, and real-time streaming architectures, ensuring data is processed efficiently and reliably.

Core Features & Use Cases

ETL/ELT Pipeline Design: Create efficient data ingestion and transformation workflows using tools like Airflow.
Data Warehousing: Design and implement scalable data warehouse schemas (star/snowflake).
Streaming Architectures: Build real-time data processing systems with Kafka or Kinesis.
Use Case: Design an Airflow DAG to ingest daily sales data from multiple sources, transform it, and load it into a Snowflake data warehouse, including data quality checks.

Quick Start

Use the data-engineer skill to design an Airflow DAG for processing daily sales data from CSV files.

Frequently Asked Questions about data-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I build an ETL pipeline using Airflow for daily data ingestion?▼

To build an ETL pipeline with Airflow, you design a DAG to ingest daily data from multiple sources, apply transformations, and load it into a data warehouse like Snowflake. The workflow includes implementing data quality checks and idempotent operations.

What is the best way to design a data warehouse schema for scalability?▼

The best way to design a scalable data warehouse schema is to implement star or snowflake schemas. This approach ensures efficient querying and maintainability while focusing on data governance and cost optimization for cloud data services.

How does incremental processing work in Spark data engineering?▼

Incremental processing in Spark works by processing only new or changed data rather than the entire dataset. It implements schema-on-read/write and idempotent operations to ensure data lineage, scalability, and maintainability across pipelines.

Can I use Kafka for real-time streaming architectures and data lineage tracking?▼

Yes, you can use Kafka to build real-time streaming architectures. It integrates with data lineage and monitoring systems to ensure data is processed efficiently and reliably, maintaining scalability and data quality across streaming pipelines.

Do I need schema-on-read or schema-on-write for cloud data services?▼

Choosing schema-on-read or schema-on-write depends on your processing needs. Schema-on-read offers flexibility for raw data ingestion, while schema-on-write enforces data quality early, both optimizing cost and governance for cloud data services.

Why does my Airflow DAG fail during data transformation and loading?▼

An Airflow DAG may fail during data transformation if idempotent operations are not implemented correctly or data quality checks fail. Ensuring proper schema-on-read/write and monitoring for data lineage helps maintain pipeline reliability.