etl

Build idempotent ETL pipelines with incremental extraction and schema validation.

Updated Feb 6, 2026

Search Tags:#orchestration #idempotent #etl #error-handling #data-pipelines #data-quality #incremental-loading

Ready to run

npx skills add https://github.com/pointware/custom-marketplace --skill etl

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: etl
Source: https://github.com/pointware/custom-marketplace/tree/main/claude-plugins/data-toolkit/skills/etl
Command: npx skills add https://github.com/pointware/custom-marketplace --skill etl

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This skill helps teams build reliable, auditable ETL pipelines that maintain data quality across ingestion, transformation, and load stages.

Core Features & Use Cases

Idempotent pipelines that can be retried safely without duplicating data.
Incremental extraction to minimize data transfer and latency.
Schema validation and data quality checks at ingestion.
Orchestrated workflows with clear checkpoints and error handling.
Observability through logging, metrics, and lineage documentation.
Use Case: Process daily transactional data from source systems into a data lake or warehouse with alerts for failures.

Quick Start

Set up a daily ETL job to ingest new records from the source database into the data warehouse, applying schema validation and quality checks along the way.

Frequently Asked Questions about etl

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I build idempotent ETL pipelines that can be safely retried?▼

Idempotent ETL pipelines use unique identifiers and state tracking to ensure retries produce identical results without duplicating data. This skill enforces idempotency through checksums, watermarks, and transactional writes so failed jobs can restart from the last successful checkpoint without reprocessing.

What's the best way to validate data quality during ETL ingestion?▼

Data quality checks validate schema, data types, and business rules at ingestion before transformation. This skill applies schema validation and quality gates at the source stage, catching malformed or incomplete records early and logging violations for audit trails.

How do I set up incremental loading to reduce ETL latency and data transfer?▼

Incremental loading extracts only new or changed records since the last successful run using timestamps or change flags. This skill tracks extraction state and applies incremental logic to minimize volume, accelerate pipelines, and reduce warehouse load times.

Can I use ETL orchestration with error handling and observable logging?▼

ETL orchestration manages workflow checkpoints, retries, and dependencies across ingest, transform, and load stages. This skill provides orchestrated pipelines with error handling, comprehensive logging, metrics, and lineage documentation for full observability and recovery.

Does this approach work for both batch and streaming data workflows?▼

This skill applies to batch ingestion (daily jobs, bulk loads) and streaming workflows (real-time sources, continuous pipelines). It enforces the same idempotency, validation, and observability patterns across both paradigms to source systems, data lakes, warehouses, and analytics platforms.