skills.rest
mtsatryan
mtsatryanCOMMUNITY
Explore all skills by this author →
GitHub

data-engineer

Develop end-to-end data pipelines for batch and real-time analytics workloads.

1|Updated Apr 23, 2026
Search Tags:#etl#airflow#kafka#spark#data-pipeline#big-data#cloud-platforms
One-click install
npx skills add https://github.com/mtsatryan/openclaw-ai-agents --skill data-engineer-mtsatryan
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-engineer
Source: https://github.com/mtsatryan/openclaw-ai-agents/tree/main/data-engineer
Command: npx skills add https://github.com/mtsatryan/openclaw-ai-agents --skill data-engineer-mtsatryan

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Data engineers face the challenge of designing and maintaining scalable data infrastructures and end-to-end pipelines that can handle batch and streaming workloads. This skill provides a comprehensive framework covering data pipeline development, big data technologies, and robust data storage solutions, with governance and quality considerations to ensure reliable analytics.

Core Features & Use Cases

  • Data Pipeline Development: ETL/ELT design, real-time and batch processing, data validation, error handling, and orchestration.
  • Big Data Technologies: Spark, Kafka, Airflow, Beam, Hadoop ecosystem, and data tooling across cloud providers.
  • Data Storage & Architecture: Data warehouses, data lakes, and databases with lineage and governance practices.
  • Use Case: Build a scalable analytics platform that ingests raw data, processes it in real time, stores results in a warehouse, and provides quality checks.

Quick Start

Describe your data pipeline requirements and run the Data Engineer skill to scaffold a scalable ETL/ELT workflow.

Frequently Asked Questions about data-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a scalable ETL pipeline for both batch and real-time data processing?▼

To build a scalable ETL pipeline, you scaffold an architecture using Spark for batch processing and Kafka for real-time streaming, orchestrated by Airflow to handle data validation, error handling, and analytics workloads.

What's the best way to design a data architecture that handles streaming and batch workloads?▼

Designing a scalable data architecture involves using data lakes and warehouses for storage, with Spark and Kafka processing batch and streaming workloads, ensuring reliable analytics through governance and data quality checks.

Can I use Airflow to orchestrate ETL workflows across multi-cloud and on-prem data stores?▼

Yes, you can use Airflow to orchestrate ETL workflows across multi-cloud and on-prem deployments, coordinating data pipelines that ingest raw data, process it, and store results in cloud data stores with quality monitoring.

How do I implement data quality and governance checks in a big data pipeline?▼

Implement data quality and governance in big data pipelines by applying validation, lineage tracking, and monitoring frameworks within your ETL design to ensure reliable analytics across Spark, Kafka, and data warehouse environments.

Does this approach support ETL development for organizations building data platforms on cloud platforms?▼

Yes, this approach supports ETL development for organizations building data platforms by providing a framework for cloud data stores, big data technologies, and scalable infrastructure tailored to analytics workloads.

When should I use Kafka instead of Spark for my data pipeline requirements?▼

Use Kafka for real-time streaming data ingestion and event-driven pipelines, while Spark handles large-scale batch processing and analytics transformations, with Airflow orchestrating both within your ETL design.

Related Skills

Explore More Data & Analytics Skills→
alex-quan-001alex-quan-001

o11y-data-export

Automate TiDB Cloud observability data exports to S3 storage.

Community
Intermediate
weirdgmeweirdgme
4

platform-engineering

Design and implement scalable platform engineering patterns for internal developer platforms.

Community
Advanced
camilooscargbaptistacamilooscargbaptista
7

data-engineering

Assess data pipelines, ETL/ELT processes, and governance across batch and streaming systems.

Community
Advanced
IndustrialIndustrial

data-engineer

Design scalable data pipelines and warehouses with Apache Spark, dbt, and Airflow.

Community
Advanced