skills.rest
KaiBoo404
KaiBoo404COMMUNITY
Explore all skills by this author →
GitHub

data-engineer

Design scalable batch and streaming data pipelines across cloud platforms.

Updated Aug 23, 2026
Search Tags:#design#governance#etl#data-pipelines#lakehouse#data-architecture#data-warehousing
One-click install
npx skills add https://github.com/KaiBoo404/agent-skills-with-project-template --skill data-engineer-kaiboo404
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-engineer
Source: https://github.com/KaiBoo404/agent-skills-with-project-template/tree/main/.agents/skills/data-engineer
Command: npx skills add https://github.com/KaiBoo404/agent-skills-with-project-template --skill data-engineer-kaiboo404

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Data engineers often struggle to design scalable pipelines across batch and streaming workloads, leading to brittle deployments and governance gaps.

Core Features & Use Cases

  • Designing batch and streaming data pipelines
  • Building data warehouses and lakehouse architectures
  • Governance, data quality, and lineage
  • Use Case: Migrating analytics to a lakehouse with clear sources, contracts, and orchestration to ensure reliable end-to-end data flows.

Quick Start

Describe a scalable data pipeline architecture for batch and streaming workloads.

Frequently Asked Questions about data-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I design scalable data pipelines for batch and streaming workloads?▼

To design scalable data pipelines, define your sources, SLAs, and ingestion methods first. Apply transformation, validation, and monitoring logic to ensure reliable end-to-end data flows across batch and streaming workloads.

What is a lakehouse architecture and when do I need it for data warehousing?▼

A lakehouse architecture combines data warehousing and data lakes. You need it when migrating analytics to unify batch and streaming pipelines, ensuring clear data contracts, governance, and lineage tracking.

How do I ensure data quality and governance across cloud platforms?▼

Ensure data quality and governance by implementing validation rules and lineage tracking across your cloud data pipelines. Apply governance frameworks to monitor data flows and maintain reliable end-to-end architectures.

Can I use this approach to optimize data pipeline costs and monitor SLAs?▼

Yes, you can optimize data pipeline costs and monitor SLAs by designing architectures that incorporate cost optimization and monitoring. Define clear sources, contracts, and orchestration to ensure reliable end-to-end data flows.

What is the best way to migrate analytics to a modern lakehouse architecture?▼

The best way to migrate analytics to a lakehouse architecture is establishing clear sources, data contracts, and orchestration. This ensures reliable end-to-end data flows and maintains governance across batch and streaming pipelines.

Related Skills

Explore More Data & Analytics Skills→
qa-amanqa-aman
19

pipeline-design-doc

Document data pipeline architectures with data flows, contracts, and governance artifacts.

Community
Advanced
wuyyyyyouwuyyyyyou

recipe-watch-drive-changes

Subscribe to Google Drive change notifications for files and folders.

Community
Intermediate
camilooscargbaptistacamilooscargbaptista
7

data-engineering

Assess data pipelines, ETL/ELT processes, and governance across batch and streaming systems.

Community
Advanced
bcastelinobcastelino
2

data-engineer

Design and implement scalable batch or streaming data pipelines.

Community
Advanced