skills.rest
bennybennison
bennybennisonCOMMUNITY
Explore all skills by this author →
GitHub

raw-store-patterns

Store raw API responses as immutable JSON records with fetched_at timestamps.

Updated Apr 11, 2026
Search Tags:#retention#medallion#audit-trail#data-ingestion#raw-store#parse-on-demand
One-click install
npx skills add https://github.com/bennybennison/agent-toolkit --skill raw-store-patterns
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: raw-store-patterns
Source: https://github.com/bennybennison/agent-toolkit/tree/main/skills/raw-store-patterns
Command: npx skills add https://github.com/bennybennison/agent-toolkit --skill raw-store-patterns

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Many integrations provide data that cannot be reliably re-fetched or have evolving schemas, forcing either brittle parsing at ingestion or loss of original context; this pattern solves the problem by capturing immutable raw responses and deferring extraction until access patterns justify structure.

Core Features & Use Cases

  • Immutable Raw Storage: Store full API response payloads with source identifiers and fetched timestamps to preserve the original data for auditing and reprocessing.
  • Parse-on-Demand: Delay JSON parsing until a value is needed, avoiding lost data when parsing fails and enabling schema evolution handling.
  • Core Table Derivation: Create structured core tables derived from raw rows when queries, joins, or indexing needs emerge.
  • Retention & Cleanup: Apply configurable retention windows and safe cleanup of processed records using processed_at and fetched_at markers.
  • Use Case: Ideal for ingesting data from customer systems, rate-limited third-party APIs, or any source where you might only obtain a single pull and must retain an audit trail.

Quick Start

Ingest API responses into raw_{source}_{entity} with payload, source_id, and fetched_at, then extract a core table when you identify frequent query patterns.

Frequently Asked Questions about raw-store-patterns

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
What is the raw store pattern for API data ingestion?▼

The raw store pattern stores full API response payloads as immutable JSON records with source identifiers and fetched timestamps. This preserves original source data for auditing and reprocessing before any structured parsing occurs.

How do I store API responses when the source schema evolves frequently?▼

Apply a parse-on-demand approach by storing raw API responses as immutable JSON records in a raw table. Delay JSON extraction until specific values are needed, preventing lost data when parsing fails and accommodating schema evolution.

What's the best way to retain API data from rate-limited third-party systems?▼

Capture non-repullable API responses in immutable raw tables with fetched_at timestamps. Apply configurable retention windows and safe cleanup of processed records using processed_at markers to manage storage while maintaining audit trails.

When should I derive core tables from raw JSON records?▼

Derive structured core tables from raw rows when frequent query patterns, joins, or indexing needs emerge. This defers extraction until access patterns justify the structure, avoiding brittle parsing at ingestion time.

Does the raw store pattern work for audit trails with non-repullable customer systems?▼

Yes, storing immutable raw API responses is ideal for customer systems where you only obtain a single pull. It preserves original payloads with source identifiers and timestamps to satisfy audit trail requirements.

How do I structure raw tables for API data ingestion?▼

Ingest API responses into raw_{source}_{entity} tables containing the payload, source_id, and fetched_at fields. Establish indexes for lookup performance, then extract a core table when frequent query patterns are identified.

Related Skills

Explore More Data & Analytics Skills→
servitolaservitola
12

airtable

Read and write Airtable bases, tables, and records via REST API with curl.

Community
Intermediate
kthornkthorn
118

Checking ChEMBL for Structured SAR Data

Query ChEMBL by DOI to retrieve curated bioactivity data counts and metadata.

Community
Basic
erichowenserichowens
181

data-pipeline-engineer

Automate ETL/ELT pipeline design, validation, and operation across medallion architectures.

Community
Advanced
borgheiborghei
467

senior-data-engineer

Automate scalable data pipeline design and warehousing with Spark, Airflow, and Snowflake.

Community
Advanced