data-engineering

Design and implement scalable data pipelines with Apache Spark and dbt workflows.

Updated May 4, 2026

Search Tags:#ETL #dbt #data engineering #Spark #data warehouse #Airflow #ELT #data lake

Ready to run

npx skills add https://github.com/luokai25/luo-ai-skills-market --skill data-engineering-luokai25

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: data-engineering
Source: https://github.com/luokai25/luo-ai-skills-market/tree/main/09-data-and-ai%20%28by%20Luo%20Kai%29/06-data-engineering/data-engineering-expert
Command: npx skills add https://github.com/luokai25/luo-ai-skills-market --skill data-engineering-luokai25

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires spark, airflow, dbt, snowflake, bigquery, redshift, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill offers expertise in advanced data engineering, tackling the creation of data pipelines, ETL/ELT processes, and leveraging tools such as Apache Spark, Airflow, dbt, and various data warehouses and lakes.

Core Features & Use Cases

Expert Data Engineering: Offers expert-level advice and implementation for data pipelines, ETL/ELT processes, and tools like Spark, Airflow, and dbt.
Data Warehouse and Lake Support: Guidance for building data warehouses like Snowflake, BigQuery, Redshift, and working with data lakes.
Use Case: Ideal for a user seeking to optimize a complex data pipeline involving batch or streaming operations and needing advice on partitioning strategies and data quality.

Quick Start

To analyze and optimize your data pipeline, start by running the skill with the command: analyze-pipeline my-pipeline-definition.yml.

Frequently Asked Questions about data-engineering

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I design a data pipeline that handles both batch and streaming data processing?▼

Designing a data pipeline for batch and streaming data processing requires partitioning strategies and tools like Apache Spark to manage throughput, while leveraging Airflow to orchestrate the workflow and ensure robust data quality across the pipeline.

What is the best way to optimize ETL processes using dbt and Snowflake?▼

Optimizing ETL processes with dbt and Snowflake involves defining modular SQL transformations in dbt to build a structured data warehouse, utilizing Snowflake's scalable compute resources to execute ELT workflows efficiently and maintain data quality.

Can I use Apache Airflow to orchestrate ELT workflows into a BigQuery data warehouse?▼

Yes, Apache Airflow can orchestrate ELT workflows into a BigQuery data warehouse by scheduling Directed Acyclic Graphs (DAGs) that trigger data extraction, execute loading operations, and automate transformations to manage your data infrastructure.

When should I choose Redshift over BigQuery for building a data warehouse?▼

Choosing Redshift over BigQuery for building a data warehouse depends on your existing infrastructure, where Redshift offers tight integration with AWS data lakes, while BigQuery provides serverless scaling for ELT processes and analytics.

Why does my Spark data pipeline fail when processing large datasets from a data lake?▼

A Spark data pipeline processing large datasets from a data lake often fails due to inadequate partitioning strategies or memory allocation, requiring optimization techniques like data repartitioning and cluster resource tuning to handle the batch load.

Do I need dbt if I am already using Apache Spark for data engineering?▼

You need dbt alongside Apache Spark when your data engineering requires structured ELT transformations, as dbt handles SQL-based transformations and data warehouse modeling, while Spark manages heavy data processing and data lake integration.