data-infrastructure-at-scale

Plan scalable data infrastructure from MVP to production.

11|1|Updated Jan 16, 2026

Search Tags:#caching #ETL #streaming #pipelines #scale #data-infrastructure

Ready to run

npx skills add https://github.com/sunnypatneedi/claude-starter-kit --skill data-infrastructure-at-scale

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: data-infrastructure-at-scale
Source: https://github.com/sunnypatneedi/claude-starter-kit/tree/main/skills/engineering/data-infrastructure-at-scale
Command: npx skills add https://github.com/sunnypatneedi/claude-starter-kit --skill data-infrastructure-at-scale

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Build scalable data infrastructure that grows from MVP to production, enabling you to handle increasing data volumes, throughput, and complexity without rewrites.

Core Features & Use Cases

Architecture planning: design scalable pipelines, storage, and compute for growth.
Data movement & storage: choose between caches, databases, message queues, and data lake architectures.
Migration & modernization: plan staged migrations, sharding, and read/write patterns to evolve systems safely.
Use Case: Imagine migrating an on-prem data warehouse to a cloud-native, multi-region setup with real-time analytics.

Quick Start

Use the data-infrastructure-at-scale skill to draft a high-level architecture, identify bottlenecks, and propose a phased migration plan.

Frequently Asked Questions about data-infrastructure-at-scale

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I scale data infrastructure from an MVP to production without rewriting pipelines?▼

To scale data infrastructure from MVP to production, plan staged migrations using sharding, replication, and caching, while evolving ETL/ELT pipelines to handle increasing data volumes safely. This prevents full system rewrites by designing distributed systems architectures early.

What's the best way to select databases and caches for high-throughput pipelines?▼

Selecting databases and caches for high-throughput pipelines requires analyzing read/write patterns and planning capacity across stages from prototype to distributed systems. Compare data lake architectures against message queues to match your specific throughput requirements.

How do I plan a migration to a distributed data lake architecture?▼

Plan a migration to a distributed data lake architecture by identifying bottlenecks in your current setup, then proposing a phased migration plan that introduces sharding and read/write separation to evolve systems safely.

Can I use this approach for both real-time streaming analytics and ETL pipelines?▼

Yes, you can use this approach for both real-time streaming analytics and ETL pipelines by integrating message queues and caching layers into your data infrastructure. This supports scalable data movement across prototype and production environments.

When do I need sharding and replication in my data infrastructure?▼

You need sharding and replication in your data infrastructure when scaling beyond a single node's capacity for read/write throughput. These mechanisms distribute load and ensure high availability as data volumes grow from MVP to production.