starrocks-realtime-modeling
CommunityBuild low-latency StarRocks upsert pipelines
Software Engineering#cdc#starrocks#materialized view#routine load#primary key upsert#real-time modeling#flink connector
Authorivanshamaev
Version1.0.0
Installs0
System Documentation
What problem does it solve?
Real-time data modeling in StarRocks for mutable data needs a reliable way to ingest Kafka/Flink/CDC changes, keep dimensions and facts current, and power near-real-time BI with predictable freshness.
Core Features & Use Cases
- Primary Key upsert modeling: Use StarRocks Primary Key tables to support streaming upserts with delete-bitmaps and mutable facts/dimensions.
- Aggregate Key & async MV pre-aggregation: Use Aggregate Key tables for immediate counter-style merges, and async materialized views for flexible dashboard aggregation refreshed on a short cadence.
- Streaming ingestion patterns: Apply Routine Load for Kafka, Flink-to-StarRocks connector for stateful transformation, and CDC flows (Debezium/Flink CDC) for upsert/delete propagation.
- Operational tuning: Configure batching, concurrency, checkpointing, and MV refresh intervals to meet an end-to-end latency/SLA while balancing consistency requirements.
Quick Start
Load a Kafka topic into a StarRocks Primary Key table with Routine Load for sub-minute BI, and configure max_batch_interval_s plus concurrent readers to match your freshness SLA.
Dependency Matrix
Required Modules
None requiredComponents
Standard package💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: starrocks-realtime-modeling Download link: https://github.com/ivanshamaev/de-agent-skills/archive/main.zip#starrocks-realtime-modeling Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 471,000+ vetted skills library on demand.