skills.rest
CODE-SAURABH
CODE-SAURABHCOMMUNITY
Explore all skills by this author →
GitHub

data-engineer

Design reliable data pipelines with idempotent processing and schema validation.

2|1|Updated Jul 25, 2026
Search Tags:#data quality#dbt#etl#airflow#kafka#data pipelines#elt
One-click install
npx skills add https://github.com/CODE-SAURABH/OpenSkills --skill data-engineer-code-saurabh
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-engineer
Source: https://github.com/CODE-SAURABH/OpenSkills/tree/main/data-engineer
Command: npx skills add https://github.com/CODE-SAURABH/OpenSkills --skill data-engineer-code-saurabh

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps prevent unreliable data pipelines that silently lose, duplicate, corrupt, or misrepresent data before it reaches dashboards, models, and business decisions.

Core Features & Use Cases

  • Pipeline Architecture: Design dependable batch, ELT, streaming, CDC, Lambda, and Kappa workflows based on freshness, volume, and operational requirements.
  • Production Engineering: Build idempotent extraction, deterministic transformations, safe loads, schema contracts, lineage, retries, and failure recovery.
  • Data Platform Standards: Apply best practices for dbt models, Airflow DAGs, Kafka consumers, data warehouses, data quality checks, and observability.
  • Use Case: Use this Skill to design an incremental orders pipeline that extracts paginated API data, validates schemas, loads it idempotently into a warehouse, transforms it with dbt, and monitors freshness and quality.

Quick Start

Ask the data-engineer skill to design a production-ready pipeline for your source data, target consumers, freshness SLA, transformations, and expected failure modes.

Frequently Asked Questions about data-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build reliable data pipelines that prevent silent data loss and corruption?▼

Build reliable data pipelines by enforcing idempotent extraction, schema validation, data quality gates, and dead-letter handling to prevent silent data loss. This ensures consistency and observability across ETL workflows before data reaches dashboards.

What's the best way to design an incremental ELT pipeline using dbt and Airflow?▼

Design an incremental ELT pipeline by extracting paginated API data, validating schemas, and loading it idempotently before applying dbt transformations. Use Airflow DAGs to orchestrate retries, monitor freshness SLAs, and manage failure recovery procedures.

How do I handle schema validation and lineage tracking in streaming Kafka pipelines?▼

Handle schema validation and lineage in Kafka streaming pipelines by implementing schema contracts, dead-letter queues for failed records, and freshness alerts. This maintains data quality and consistency across consumers during CDC and streaming workflows.

When do I need idempotent processing and data quality gates in ETL workflows?▼

You need idempotent processing and data quality gates in ETL workflows when silent duplication or corruption threatens analytics. They ensure deterministic transformations, safe loads, and reproducible pipeline recovery during failure modes.

Can I use this approach for both batch data warehouses and real-time CDC workflows?▼

Yes, this approach supports both batch data warehouses and real-time CDC workflows by adapting Lambda and Kappa architectures. It scales pipeline design based on data freshness, volume, and operational requirements for reliable analytics.

Why does my data pipeline lose records during Airflow DAG retries and how do I fix it?▼

Data pipelines lose records during Airflow DAG retries when extraction lacks idempotency and deterministic transformations. Fix this by implementing schema contracts, safe loads, dead-letter handling, and documented recovery procedures.

Related Skills

Explore More Data & Analytics Skills→
qa-amanqa-aman
19

pipeline-design-doc

Document data pipeline architectures with data flows, contracts, and governance artifacts.

Community
Advanced
nrakownrakow
1

data-lineage

Trace data lineage across dbt sources, transformations, and reports.

Community
Advanced
ouakarouakar

data-engineer

Automate end-to-end data pipelines from ingestion to analytics-ready datasets.

Community
Advanced
ps-carvalhops-carvalho

data-engineering

Design ETL/ELT pipelines with validation, monitoring, and partitioning patterns.

Community
Advanced