pipeline-patterns

What problem does it solve?

This Skill helps you design and operate data pipelines that remain correct under reruns, backfills, schema changes, and production failures—preventing duplicates, silent data corruption, and blind spots in monitoring.

Core Features & Use Cases

• Idempotency first: ensures repeated runs yield identical outputs using upsert/merge semantics, dedup keys, and safe write boundaries.
• Time partitioning & chunked processing: supports scale by partitioning by event time, defining late-arrival windows, and reprocessing only the needed partitions.
• Schema evolution without breakage: distinguishes safe backward-compatible changes from breaking changes and guides rollout/versioning strategy.
• Backfill safety: provides checklists and staged execution (dry run, small slice, controlled expansion, verification) to avoid corrupting historical data.
• Built-in observability & data quality: defines required metrics/alerts, logging contracts, and test expectations at stage boundaries.
• Dependency management: enforces explicit upstream dependencies and DAG hygiene to avoid stale-data cascades.
• Use case: when a user-events pipeline must be backfilled for a month after a transformation bug, this Skill guides the rerun plan, schema-safe rollout, and verification steps so downstream reporting stays trustworthy.

Quick Start

Ask the agent to apply the pipeline patterns to your planned ETL/ELT job, including idempotency, time partitioning, schema evolution rules, a backfill plan, and the exact observability metrics and alerts to implement.