data-engineer

Name: data-engineer
Availability: InStock
Author: hainamchung

Design scalable data pipelines for batch and streaming workloads.

54|18|Updated Jan 21, 2026

Search Tags:#streaming #dbt #etl #airflow #data-pipelines #data-warehousing #apache-spark

Ready to run

npx skills add https://github.com/hainamchung/agent-assistant --skill data-engineer-hainamchung

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: data-engineer
Source: https://github.com/hainamchung/agent-assistant/tree/main/skills/data-engineer
Command: npx skills add https://github.com/hainamchung/agent-assistant --skill data-engineer-hainamchung

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Data teams spend substantial time designing, implementing, and maintaining end-to-end data pipelines across diverse tools and environments. This skill provides a structured approach to building scalable data pipelines, modern data architectures, and reliable analytics infrastructure, reducing toil and accelerating delivery.

Core Features & Use Cases

End-to-end data pipeline design: ingestion, transformation, validation, and governance across batch and streaming workloads.
Modern data stack integration: data warehouses, lakehouses, cloud storage, and orchestration via tools like Spark, dbt, and Airflow.
Real-world scenarios: dashboards, ML feature pipelines, and data quality monitoring for production systems.

Quick Start

Create a scalable end-to-end data pipeline from S3 to Snowflake with dbt transformations and Airflow orchestration.

Frequently Asked Questions about data-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I build a scalable data pipeline from S3 to Snowflake with dbt and Airflow?▼

Building scalable data pipelines requires structured architecture design across ingestion, transformation, validation, and orchestration stages. This approach reduces manual toil by automating batch and streaming workloads, ensuring reliable data delivery to analytics platforms.

How does Apache Spark handle both batch and streaming data workloads?▼

Apache Spark processes batch and streaming data workloads by applying unified transformation and validation logic. This supports enterprise data platforms in handling both historical data processing and real-time ingestion within a single scalable data pipeline architecture.

Can I use dbt for data validation and transformation in enterprise data warehouses?▼

Yes, you can use dbt for data validation and transformation within enterprise data warehouses. It structures transformations to support production-grade requirements, ensuring data quality monitoring and reliable analytics infrastructure across batch processing workflows.

What is the best way to orchestrate data pipelines with Airflow for production systems?▼

The best way to orchestrate data pipelines with Airflow involves implementing structured error handling and monitoring for production systems. This ensures reliable execution of ingestion and transformation tasks across batch and streaming data workloads.

How do I design cost-optimized data architectures for streaming and batch ingestion?▼

Designing cost-optimized data architectures for streaming and batch ingestion requires strategic tool selection and structured pipeline design. This approach minimizes operational costs while maintaining scalable data processing and production-grade reliability.

When should I not use Apache Spark for data pipeline transformations?▼

You should reconsider using Apache Spark for data pipeline transformations when simpler orchestration tools can achieve the required batch processing without the overhead of distributed computing, or when cost optimization outweighs the need for massive scalability.