agency-data-engineer

Name: agency-data-engineer
Availability: InStock
Author: bomberoxenviosdosruedas

Build and manage ETL/ELT data pipelines and lakehouse architectures.

1|Updated May 5, 2026

Search Tags:#data pipeline #data engineering #ETL/ELT #Apache Spark #lakehouse architecture

Ready to run

npx skills add https://github.com/bomberoxenviosdosruedas/01EnviosDosRueda --skill agency-data-engineer-bomberoxenviosdosruedas

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: agency-data-engineer
Source: https://github.com/bomberoxenviosdosruedas/01EnviosDosRueda/tree/main/.agents/workflows/agency-data-engineer
Command: npx skills add https://github.com/bomberoxenviosdosruedas/01EnviosDosRueda --skill agency-data-engineer-bomberoxenviosdosruedas

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires spark, dbt, kafka, flink, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides expert data engineering capabilities to build and manage reliable data pipelines, lakehouse architectures, and scalable data infrastructure.

Core Features & Use Cases

Data Pipeline Engineering: Design and build ETL/ELT pipelines that are idempotent, observable, and self-healing.
Data Platform Architecture: Architect cloud-native data lakehouses on major cloud platforms.
Data Quality & Reliability: Implement data quality checks, schema validation, and anomaly detection.
Streaming & Real-Time Data: Build event-driven pipelines with Apache Kafka and stream processing with Apache Flink.

Quick Start

Use the agency-data-engineer skill to build a data pipeline for your project.

Frequently Asked Questions about agency-data-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I build an ETL pipeline with Apache Spark that handles schema validation and data quality checks?▼

To build a reliable ETL pipeline with Apache Spark, implement idempotent processing logic, schema validation, and anomaly detection. This approach ensures data quality and consistency during extraction, transformation, and loading operations.

What is lakehouse architecture and when do I need it for my data infrastructure?▼

Lakehouse architecture combines the scalability of data lakes with the structured querying of data warehouses. You need it when building cloud-native data platforms that require reliable data processing, schema enforcement, and scalable infrastructure.

Can I use Apache Kafka and Flink together for real-time event-driven data pipelines?▼

Yes, you can use Apache Kafka and Flink together for real-time event-driven data pipelines. Kafka handles event streaming and ingestion, while Flink provides stream processing capabilities to build scalable, self-healing data infrastructure.

Does dbt work well for ETL and ELT data pipeline engineering?▼

dbt works effectively for ELT data pipeline engineering by applying SQL-based transformations within the data warehouse. It supports idempotent and observable pipeline design, ensuring reliable data processing and infrastructure management.

What's the best way to architect a cloud-native data lakehouse that is observable and self-healing?▼

Architecting a cloud-native data lakehouse requires combining scalable storage with robust processing frameworks like Apache Spark. Implementing data quality checks, anomaly detection, and idempotent pipelines ensures the infrastructure remains observable and self-healing.

Why does my streaming data pipeline fail to maintain data quality during high-throughput ingestion?▼

Streaming data pipelines often fail data quality checks during high-throughput ingestion due to missing schema validation and anomaly detection. Implementing robust event-driven processing with Apache Kafka and Flink resolves these reliability issues.