data-engineer

Design data pipelines, ETL/ELT processes, and data platform architectures.

Updated May 4, 2026

Search Tags:#data pipeline #data engineering #stream processing #ETL/ELT #data lake

Ready to run

npx skills add https://github.com/luokai25/luo-ai-skills-market --skill data-engineer-luokai25

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: data-engineer
Source: https://github.com/luokai25/luo-ai-skills-market/tree/main/09-data-and-ai%20%28by%20Luo%20Kai%29/14-other-ai/data-engineer
Command: npx skills add https://github.com/luokai25/luo-ai-skills-market --skill data-engineer-luokai25

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill provides specialized assistance for designing, building, and optimizing data pipelines, ETL/ELT processes, and data infrastructure, ensuring high performance, scalability, and cost efficiency.

Core Features & Use Cases

Data Pipeline Design: Expertly design scalable data platforms and pipelines.
ETL/ELT Development: Implement robust ETL/ELT processes with efficient data flow and transformation.
Data Lake/Warehouse Design: Create optimized storage architectures for data lakes and warehouses.
Stream Processing: Develop real-time data processing solutions with strong state management and error handling.
Use Case: For a company looking to implement a new data pipeline for handling large-scale data analytics, this Skill would assist in designing the architecture, developing the ETL processes, and setting up monitoring and governance.

Quick Start

Use the data-engineer skill to design a new data pipeline for a large-scale data analytics project.

Frequently Asked Questions about data-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I design a scalable data pipeline for large-scale analytics?▼

Designing scalable data pipelines requires architecting robust data platforms with efficient data flow, transformation, and storage strategies. This involves implementing optimized ETL/ELT processes alongside proper orchestration and monitoring for high performance and cost efficiency.

What is the best way to build real-time stream processing with strong state management?▼

Building real-time stream processing involves developing solutions with strong state management and error handling. It requires knowledge of big data tools and cloud platforms to ensure reliable, real-time data ingestion and processing within your data infrastructure.

How do I optimize data lake and data warehouse storage architecture?▼

To optimize data lake and data warehouse storage architecture, you must create optimized storage structures that ensure high performance and scalability. This requires evaluating processing patterns and storage strategies to handle large-scale data analytics efficiently.

Does this data engineering approach support ETL and ELT development?▼

Yes, this data engineering approach supports both ETL and ELT development. It provides comprehensive solutions for implementing robust ETL/ELT processes, ensuring efficient data flow and transformation within your data pipelines and infrastructure.

When do I need disaster recovery and monitoring for data infrastructure?▼

You need disaster recovery and monitoring for data infrastructure when handling large-scale data analytics to ensure high availability and performance. Setting up monitoring and governance helps maintain pipeline reliability and protects against system failures.