skills.rest
HemantSudarshan
HemantSudarshanCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

data-engineer

Design scalable data pipelines for batch and streaming workloads.

Updated Mar 17, 2026
Search Tags:#cloud#etl#data-pipelines#lakehouse#schema-evolution#data-validation#data-architecture
One-click install
npx skills add https://github.com/HemantSudarshan/Dhumichatbot --skill data-engineer-hemantsudarshan
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-engineer
Source: https://github.com/HemantSudarshan/Dhumichatbot/tree/main/skills/02-data/data-engineer
Command: npx skills add https://github.com/HemantSudarshan/Dhumichatbot --skill data-engineer-hemantsudarshan

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Data engineers face the challenge of designing robust, scalable data pipelines and analytics infrastructure that can handle both batch and real-time workloads across cloud platforms, with governance and cost considerations.

Core Features & Use Cases

  • Design and implement end-to-end data pipelines (ingestion, transformation, validation) for batch and streaming workloads.
  • Manage data contracts, SLAs, and governance to enable trusted analytics across teams.
  • Build lakehouse architectures using modern tools (Delta Lake, dbt, Airflow) on cloud data platforms for scalable analytics.

Quick Start

Define sources, SLAs, and data contracts to start building a scalable data pipeline.

Frequently Asked Questions about data-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I design scalable data pipelines for batch and streaming workloads?▼

Build lakehouse architectures using Delta Lake, dbt, and Airflow on cloud data platforms to combine scalable storage with reliable analytics. This approach supports schema evolution, data validation, and governance for trusted analytics across teams.

What's the best way to manage schema evolution and data validation in ETL pipelines?▼

Manage schema evolution and data validation by applying data contracts and automated validation rules within your ETL pipelines. This ensures reliable transformations and prevents downstream analytics failures when source data structures change.

Can I use Airflow and dbt together to build analytics infrastructure on cloud platforms?▼

Yes, you can use Airflow for pipeline orchestration and dbt for data transformations to build robust analytics infrastructure on cloud platforms. This combination supports scalable lakehouse architectures with reliable data governance and cost efficiency.

Why does my data architecture struggle with cost efficiency and governance?▼

Your data architecture may struggle with cost efficiency and governance if it lacks defined data contracts, SLAs, and modern lakehouse structures. Implementing tools like Delta Lake and dbt on cloud platforms helps enforce governance and optimize costs.

Related Skills

Explore More Data & Analytics Skills→
JaviMontanoJaviMontano

metodologia-data-engineering

Automate data platform architecture design for ingestion, storage, quality, and observability.

Community
Advanced
akillnessakillness
40

data-analysis

Analyze CSV, JSON, and SQL datasets with Python pandas workflows.

Community
Intermediate
borgheiborghei
467

senior-ml-engineer

Provide production-grade ML engineering guidance and templates for end-to-end projects.

Community
Advanced
AradotsoAradotso
5

amee-joshi-data-engineering-portfolio

Provides Azure data engineering patterns with Medallion architecture and ETL/ELT pipelines.

Official
Advanced