data-engineer

Orchestrate end-to-end data ingestion, transformation, and loading with data contracts and lineage.

22|2|Updated Mar 24, 2026

Search Tags:#validation #data #idempotent #pipeline #etl #backfill #lineage

Ready to run

npx skills add https://github.com/jshsakura/awesome-opencode-skills --skill data-engineer-jshsakura

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: data-engineer
Source: https://github.com/jshsakura/awesome-opencode-skills/tree/main/skills/data-engineer
Command: npx skills add https://github.com/jshsakura/awesome-opencode-skills --skill data-engineer-jshsakura

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Data engineering ensures correctness, reliability, and lineage for production data pipelines across ingestion, transformation, and loading stages.

Core Features & Use Cases

End-to-end pipeline design, validation, and observability to prevent data contracts violations.
Data quality controls, schema validation, and idempotent reprocessing across batch and streaming workloads.
Use Case: Build a trusted data ingest -> transform -> load flow for a data warehouse with backfill safety and lineage tracing.

Quick Start

Create a minimal, safe ETL pipeline that ingests CSV data, applies schema validation, and loads into a data warehouse with observable lineage.

Frequently Asked Questions about data-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I build an ETL pipeline with schema validation and lineage tracing?▼

To build an ETL pipeline with schema validation and lineage tracing, orchestrate end-to-end data ingestion, transformation, and loading stages while enforcing data contracts. This ensures observable quality metrics and prevents data contract violations across warehouse integrations.

What is idempotent processing and when do I need it for data pipelines?▼

Idempotent processing ensures running a data pipeline multiple times produces the same result without duplicating records. You need it for backfill scenarios and batch reprocessing to maintain data integrity and prevent duplicate loads in analytics environments.

How do I handle backfill scenarios safely in a data warehouse integration?▼

Handle backfill scenarios safely by applying idempotent processing and schema validation during data warehouse integrations. This reprocessing approach ensures historical data loads maintain correctness and preserve observable lineage without overwriting current production records.

Can I use this approach for both streaming and batch data workloads?▼

Yes, this approach applies to both streaming and batch data workloads. It orchestrates end-to-end ingestion, transformation, and loading while applying data quality controls, schema validation, and error handling consistently across both workload types.

What's the best way to prevent data contracts violations in production pipelines?▼

The best way to prevent data contracts violations is implementing end-to-end pipeline validation and observability. Applying schema validation, data quality controls, and lineage tracing across ingestion and transformation stages catches errors before loading.

Why does my data pipeline fail during schema validation and how do I add error handling?▼

Data pipelines fail during schema validation when incoming data violates predefined data contracts. Add error handling and observable quality metrics across your transformation stages to catch, log, and manage these validation failures before loading.