Build idempotent ETL pipelines with incremental extraction and schema validation.
npx skills add https://github.com/pointware/custom-marketplace --skill etl
Please help me install this Agent Skill. Skill: etl Source: https://github.com/pointware/custom-marketplace/tree/main/claude-plugins/data-toolkit/skills/etl Command: npx skills add https://github.com/pointware/custom-marketplace --skill etl
This skill helps teams build reliable, auditable ETL pipelines that maintain data quality across ingestion, transformation, and load stages.
Set up a daily ETL job to ingest new records from the source database into the data warehouse, applying schema validation and quality checks along the way.
High-intent search queries and answers about installing and using this skill.
Idempotent ETL pipelines use unique identifiers and state tracking to ensure retries produce identical results without duplicating data. This skill enforces idempotency through checksums, watermarks, and transactional writes so failed jobs can restart from the last successful checkpoint without reprocessing.
Data quality checks validate schema, data types, and business rules at ingestion before transformation. This skill applies schema validation and quality gates at the source stage, catching malformed or incomplete records early and logging violations for audit trails.
Incremental loading extracts only new or changed records since the last successful run using timestamps or change flags. This skill tracks extraction state and applies incremental logic to minimize volume, accelerate pipelines, and reduce warehouse load times.
ETL orchestration manages workflow checkpoints, retries, and dependencies across ingest, transform, and load stages. This skill provides orchestrated pipelines with error handling, comprehensive logging, metrics, and lineage documentation for full observability and recovery.
This skill applies to batch ingestion (daily jobs, bulk loads) and streaming workflows (real-time sources, continuous pipelines). It enforces the same idempotency, validation, and observability patterns across both paradigms to source systems, data lakes, warehouses, and analytics platforms.
manutejEngineer dbt models that transform raw data into analytics-ready datasets.
williamzujkowskiValidate dataset completeness, accuracy, and consistency with Great Expectations.
TheGrowthExponentBuild and orchestrate Databricks ETL pipelines with Delta Live Tables.
HouseGarofaloDesign, implement, and validate data pipelines with Spark, Airflow, and Kafka.
