building-deterministic-data-pipelines

What problem does it solve?

This Skill helps you eliminate run-to-run drift in shared data pipelines so outputs stay byte-for-byte identical, auditable, and safe for downstream use.

Core Features & Use Cases

• Canonical output rules: Enforces stable JSON, JSONL, CSV, and Parquet writing through sorted keys, sorted records, pinned float formatting, and LF line endings.
• Provenance tracking: Adds a sibling provenance.json with source metadata, adapter version, input hashes, output hash, and row counts for reviewer-friendly traceability.
• Drift detection: Re-runs the pipeline in CI and compares hashes to catch silent changes from code, dependencies, source schema drift, or formatting changes.
• Use cases: Ideal for ingest pipelines, ETL jobs, preprocessing steps, model-training inputs, and any artifact that must be reproducible across machines and time.

Quick Start

Ask the skill to make your shared pipeline output deterministic by identifying the format, applying canonicalization rules, writing provenance metadata, and adding a CI drift check.