data-source-pattern
CommunityStandardize data ingestion with patterns
Software Engineering#scraping#data-quality#data-pipeline#data-ingestion#seed-scripts#etl-patterns#api-ingestion
Authorexerias21
Version1.0.0
Installs0
System Documentation
What problem does it solve?
This guide provides a structured approach to ingest external data into a project's database, ensuring consistency, traceability, and scalable handling of new sources.
Core Features & Use Cases
- Pattern 1 — Discovery Pipeline (automated, scheduled): defines a cron-driven workflow that fetches external data, deduplicates, upserts into the DB, and emits metrics.
- Pattern 2 — Seed Script (one-time bulk load): offers idempotent scripts to load static datasets with safe re-run behavior.
- Pattern 3 — Direct API Ingestion (user or model generated): provides a request/endpoint approach to insert data synchronously with validation.
Quick Start
Define a data source and implement one of the three patterns to ingest it into your database.
Dependency Matrix
Required Modules
None requiredComponents
Standard package💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: data-source-pattern Download link: https://github.com/exerias21/brainstorm-toolkit/archive/main.zip#data-source-pattern Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 471,000+ vetted skills library on demand.