skills.rest
bcastelino
bcastelinoCOMMUNITY
Explore all skills by this author →
Website𝕏@cas7elinoGitHub

data-engineer

Design and implement scalable batch or streaming data pipelines.

2|Updated Feb 6, 2026
Search Tags:#streaming#data engineering#etl#data pipelines#data architecture#lakehouse#data warehousing
One-click install
npx skills add https://github.com/bcastelino/agent-skills-kit --skill data-engineer-bcastelino
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-engineer
Source: https://github.com/bcastelino/agent-skills-kit/tree/main/skills/data-engineer
Command: npx skills add https://github.com/bcastelino/agent-skills-kit --skill data-engineer-bcastelino

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill addresses the complexity of designing, building, and maintaining robust, scalable data pipelines and modern data architectures, ensuring reliable data flow and accessibility for analytics and operations.

Core Features & Use Cases

  • Pipeline Design: Architect batch or streaming data pipelines.
  • Data Warehousing/Lakehouse: Build and manage data warehouses or lakehouse architectures.
  • Data Quality & Governance: Implement data quality checks and governance policies.
  • Use Case: Design and implement a real-time streaming pipeline that processes 1 million events per second from Kafka to BigQuery, ensuring data quality and low latency.

Quick Start

Design a real-time streaming pipeline that processes 1M events per second from Kafka to BigQuery.

Frequently Asked Questions about data-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I design scalable data pipelines for high-throughput streaming data?▼

To build scalable data pipelines for streaming data, you must architect workflows that process high-throughput events from sources like Kafka to targets like BigQuery while maintaining low latency, data quality, and operational reliability.

What is the best way to architect a lakehouse solution for batch and streaming data?▼

Architecting a lakehouse solution involves building a unified data architecture that supports both batch and streaming data processing, ensuring reliable data flow, governance policies, and accessibility for analytics and operations.

Can I implement data quality and governance checks within my ETL data pipelines?▼

Yes, you can implement data quality and governance checks directly within ETL data pipelines to ensure reliable data flow, enforce governance policies, and maintain operational reliability across batch or streaming data processing workflows.

How do I build a real-time data pipeline processing events from Kafka to BigQuery?▼

Building a real-time data pipeline from Kafka to BigQuery involves designing a streaming architecture that handles high event volumes, ensures data quality, and maintains low latency for immediate analytics accessibility.

When should I choose a data warehouse over a lakehouse architecture?▼

Choosing between a data warehouse and a lakehouse architecture depends on whether your data pipelines prioritize structured batch processing or require flexible, unified batch and streaming data accessibility for analytics and operations.

Related Skills

Explore More Data & Analytics Skills→
angryracoonangryracoon

data360-prepare

Prepare Salesforce Data Cloud ingestion assets for raw source data.

Community
Advanced
fatihkanfatihkan
7

data-analytics

Analyze datasets and generate analytics reports with visuals.

Community
Advanced
imMamdouhaboammarimMamdouhaboammar

agency-devops-automator

Automate infrastructure provisioning and CI/CD pipeline development.

Community
Advanced
IndustrialIndustrial

data-engineer

Design scalable data pipelines and warehouses with Apache Spark, dbt, and Airflow.

Community
Advanced