raw-store-patterns

What problem does it solve?

Many integrations provide data that cannot be reliably re-fetched or have evolving schemas, forcing either brittle parsing at ingestion or loss of original context; this pattern solves the problem by capturing immutable raw responses and deferring extraction until access patterns justify structure.

Core Features & Use Cases

• Immutable Raw Storage: Store full API response payloads with source identifiers and fetched timestamps to preserve the original data for auditing and reprocessing.
• Parse-on-Demand: Delay JSON parsing until a value is needed, avoiding lost data when parsing fails and enabling schema evolution handling.
• Core Table Derivation: Create structured core tables derived from raw rows when queries, joins, or indexing needs emerge.
• Retention & Cleanup: Apply configurable retention windows and safe cleanup of processed records using processed_at and fetched_at markers.
• Use Case: Ideal for ingesting data from customer systems, rate-limited third-party APIs, or any source where you might only obtain a single pull and must retain an audit trail.

Quick Start

Ingest API responses into raw_{source}_{entity} with payload, source_id, and fetched_at, then extract a core table when you identify frequent query patterns.