skills.rest
ps-carvalho
ps-carvalhoCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

data-engineering

Design ETL/ELT pipelines with validation, monitoring, and partitioning patterns.

Updated Feb 22, 2026
Search Tags:#monitoring#validation#streaming#partitioning#etl#data-pipelines#elt
One-click install
npx skills add https://github.com/ps-carvalho/cortex-agents --skill data-engineering-ps-carvalho
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-engineering
Source: https://github.com/ps-carvalho/cortex-agents/tree/main/.opencode/skills/data-engineering
Command: npx skills add https://github.com/ps-carvalho/cortex-agents --skill data-engineering-ps-carvalho

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Large data projects struggle with unreliable pipelines, data quality issues, and slow iteration. This skill provides patterns and best practices to design, implement, and operate robust data pipelines.

Core Features & Use Cases

  • ETL/ELT pipeline design: Architect robust pipelines that reliably ingest, transform, and load data with validation and monitoring.
  • Data validation & schema enforcement: Ensure data quality at ingestion and during transformations, with backward-compatible evolution.
  • Streaming & partitioning patterns: Apply windowing, stateful processing, and partitioning strategies for real-time analytics and batch workloads.
  • Message queues & storage patterns: Use message queues and storage architectures to decouple components and enable reliable data flow.

Quick Start

Design and implement a data-pipeline plan for a new dataset, including ingestion, validation, and error handling.

Frequently Asked Questions about data-engineering

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build reliable data pipelines with schema enforcement and dead-letter handling?▼

To build reliable data pipelines, apply standardized ETL/ELT patterns that enforce schema validation at ingestion and route invalid records to dead-letter queues, ensuring idempotent processing and consistent data quality across batch workloads.

What is the best way to handle data validation and schema evolution in ETL pipelines?▼

Data validation in ETL pipelines is handled by enforcing schemas during ingestion and transformations, ensuring backward-compatible evolution so downstream analytics remain reliable without breaking existing data flows or processing logic.

How do I design streaming data pipelines with partitioning and stateful processing?▼

Streaming data pipelines are designed using windowing, stateful processing, and partitioning strategies to handle real-time analytics workloads, decoupled via message queues to enable scalable and reliable data flow across modern architectures.

Can I use this approach for both batch and streaming data workloads?▼

Yes, these data pipeline patterns support both batch and streaming data workloads by applying consistent partitioning strategies, validation, and monitoring to ensure reliable ingestion and transformation regardless of the processing mode.

Why do my data pipelines fail during schema changes and how do I fix it?▼

Data pipelines fail during schema changes due to rigid ingestion contracts; fix this by implementing schema enforcement with backward-compatible evolution and dead-letter handling to isolate malformed records without halting the pipeline.

Do I need message queues to decouple components in modern data architectures?▼

Message queues are needed to decouple components in modern data architectures, enabling reliable data flow between ingestion, transformation, and storage layers while supporting scalable processing for both batch and streaming workloads.

Related Skills

Explore More Data & Analytics Skills→
JaviMontanoJaviMontano

metodologia-data-engineering

Automate data platform architecture design for ingestion, storage, quality, and observability.

Community
Advanced
herdiansahherdiansah
23

business-analyst

Analyze business data to create KPI frameworks and predictive models.

Community
Advanced
arolariuarolariu
6

tasks

Manage personal task stacks and monitor automated pipeline processing queues.

Community
Advanced
sandeepmvlsandeepmvl
21

data-warehouse-integration

Design ELT pipelines syncing Rails Postgres to cloud warehouses with dbt.

Community
Advanced