skills.rest
FluxonLab
FluxonLabCOMMUNITY
Explore all skills by this author →
Website𝕏@fluxonlabGitHub

data-pipeline-review

Review data and ML pipelines for idempotency, watermarking, orchestration safety, and backfill correctness.

2|Updated Jun 1, 2026
Search Tags:#data quality#data pipeline#etl#airflow#idempotency#backfill#watermark
One-click install
npx skills add https://github.com/FluxonLab/Skillry --skill data-pipeline-review-fluxonlab
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-pipeline-review
Source: https://github.com/FluxonLab/Skillry/tree/main/plugins/data-ml-ai-engineering/skills/311-data-pipeline-review
Command: npx skills add https://github.com/FluxonLab/Skillry --skill data-pipeline-review-fluxonlab

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Data pipelines that append under retries, lose late-arriving data, or run destructive backfills silently corrupt production tables. This Skill reviews or designs batch and streaming pipelines so they are idempotent, restartable, and correct under retries and backfills. ## Core Features & Use Cases - Idempotency audit: Classifies every load as partition overwrite, MERGE/upsert, or unsafe bare append, and flags duplicate-row risks under retry. - Incremental correctness: Verifies watermark cursors use event timestamps, persist only after success, and include a late-data lookback window. - Orchestration and backfill safety: Checks Airflow, Dagster, Prefect, or dbt definitions for bounded retries, concurrency limits, deliberate catchup settings, and non-destructive parameterized backfills. - Use Case: A PR adds an Airflow DAG that loads orders hourly. Use this Skill to confirm the load uses MERGE on a stable key, the watermark persists after success, catchup is disabled, and a backfill plan targets only affected partitions. ## Quick Start Review the Airflow DAG and dbt models in this repository for idempotency, watermark handling, and backfill safety, then produce a findings report.

Frequently Asked Questions about data-pipeline-review

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I make a data pipeline idempotent?▼

Make each load a full partition overwrite, a MERGE/upsert on a stable key, or an insert into a fresh partition that is atomically swapped. A bare INSERT with retries duplicates rows, so re-running a task on the same interval must produce identical row counts.

How to handle late-arriving data in incremental loads?▼

Use a watermark based on an event or business timestamp, not wall-clock time, and re-pull a lookback window (for example two hours) so late rows are captured. Persist the watermark only after the load succeeds, and MERGE rather than append to stay idempotent.

What Airflow settings prevent duplicate pipeline writes?▼

Set bounded retries with retry_delay, max_active_runs=1 so two runs never write the same partition, and catchup=False to avoid an unintended multi-month backfill at deploy time. Set depends_on_past deliberately rather than leaving defaults.

Why does my pipeline produce duplicate rows after retries?▼

Duplicates happen when a task uses append-only INSERT and the orchestrator retries it after a partial write. Switch to MERGE/upsert on a stable key or partition overwrite so re-execution replaces rather than duplicates data.

When should I not use a full pipeline review?▼

Skip it for one-off ad-hoc queries with no scheduling or persistence. For pure data-quality assertion logic use the data-quality-validation reference, and for dataset versioning or lineage metadata only use the dataset-versioning-and-lineage reference.

Related Skills

Explore More Data & Analytics Skills→
organvm-i-theoriaorganvm-i-theoria

data-ingestion-pipeline

Build batch and streaming data ingestion pipelines with extraction, validation, deduplication, and staging.

Official
Intermediate
vtran5290vtran5290
3

Schema Guardian

Validate weekly report payloads against a fixed JSON schema.

Community
Intermediate
LeandroBenjaminLLeandroBenjaminL

data-validation

Validate data schemas and quality using Pandera and Great Expectations.

Community
Advanced
chicanoandres702chicanoandres702

data-engineering-data-pipeline

Design and orchestrate scalable batch and streaming data pipelines.

Community
Advanced