data-engineer-agent

Design, implement, and validate data pipelines with Spark, Airflow, and Kafka.

3|1|Updated Feb 2, 2026

Search Tags:#data pipeline #data engineering #etl #airflow #kafka #data warehouse #spark

Ready to run

npx skills add https://github.com/HouseGarofalo/claude-code-base --skill data-engineer-agent

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: data-engineer-agent
Source: https://github.com/HouseGarofalo/claude-code-base/tree/main/.claude/skills/data-engineer-agent
Command: npx skills add https://github.com/HouseGarofalo/claude-code-base --skill data-engineer-agent

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill automates the design, implementation, and quality assurance of data pipelines, ETL/ELT processes, and data warehousing solutions, enabling efficient data transformation and analytics infrastructure.

Core Features & Use Cases

Pipeline Design & Implementation: Creates batch and streaming data pipelines using tools like Spark, Airflow, and Kafka.
Data Warehousing: Designs and models data warehouses using star schemas.
Data Quality: Implements data quality checks and frameworks to ensure data integrity.
Use Case: Design and build a daily ETL pipeline to ingest sales data from an API, transform it using Spark, load it into a Snowflake data warehouse, and implement data quality checks to ensure accuracy.

Quick Start

Use the data-engineer-agent skill to design an ETL pipeline for daily sales data.

Frequently Asked Questions about data-engineer-agent

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I build an ETL pipeline to ingest, transform, and load data into a warehouse?▼

To build an ETL pipeline, you design and implement data ingestion, transform the data using Spark, and load it into a data warehouse like Snowflake or AWS Redshift. This Skill automates that workflow, including validation and data quality checks.

What is the best way to implement data quality checks in a Spark data pipeline?▼

Implementing data quality checks in a Spark data pipeline involves applying validation frameworks to ensure data integrity during transformation. This Skill designs and integrates these checks directly into your ETL/ELT processes to maintain accuracy.

Can I use Airflow and Kafka to design streaming data architectures?▼

Yes, you can use Airflow and Kafka to design streaming data architectures. This Skill specializes in creating streaming pipelines with Kafka and orchestrating batch processing workflows with Airflow for scalable data engineering.

Does this support data warehouse modeling with star schemas for cloud platforms?▼

Yes, this supports data warehouse modeling with star schemas for cloud platforms. It designs and models data warehouses specifically for cloud environments like Snowflake, Databricks, and AWS Redshift to structure analytics data efficiently.

How do I automate a daily batch processing workflow for sales data?▼

To automate a daily batch processing workflow for sales data, you configure a scheduled pipeline that extracts data from an API, transforms it with Spark, and loads it into your data warehouse. This Skill designs and validates these automated batch workflows.