What problem does it solve?
Creating reliable, resumable data extraction pipelines from legacy relational systems is hard because the source often lacks CDC, stable change feeds, or trustworthy timestamps.
Core Features & Use Cases
- Entity Contract first: Define canonical entities (name, PK, updated column, expected output columns) and an explicit FK graph before writing any SQL or stages.
- Snapshot + delta via content hashing: Detect new/modified/deleted rows by hashing row content between the current snapshot and the prior approved baseline.
- Layered quality gates: Enforce structural hard-fail checks (PK, uniqueness, schema coverage, empty-string FK risks) and run scored format/semantic validation plus human approval before loading.
- Stage orchestration with resume sentinels: Use numeric stage IDs and per-stage sentinel files so failed runs resume from the exact stage without redoing side effects.
- Filesystem-namespaced audit trail: Produce immutable raw/clean extracts, delta manifests, validation artifacts, review packages, and approved outputs per {entity}/{run_id}.
Quick Start
Use the bronze-extraction-pipeline skill when you are designing a Bronze/medallion extraction workflow that must be reproducible, resumable, and delta-aware from a relational source system.
