skills.rest
ivanshamaev
ivanshamaevCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

starrocks-realtime-modeling

Designs StarRocks ingestion and modeling for mutable tables via Kafka, Flink, or CDC streams.

14|1|Updated May 5, 2026
Search Tags:#cdc#starrocks#materialized view#routine load#primary key upsert#real-time modeling#flink connector
One-click install
npx skills add https://github.com/ivanshamaev/de-agent-skills --skill starrocks-realtime-modeling
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: starrocks-realtime-modeling
Source: https://github.com/ivanshamaev/de-agent-skills/tree/main/group_skills/starrocks_group_skills/starrocks_realtime_modeling
Command: npx skills add https://github.com/ivanshamaev/de-agent-skills --skill starrocks-realtime-modeling

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Real-time data modeling in StarRocks for mutable data needs a reliable way to ingest Kafka/Flink/CDC changes, keep dimensions and facts current, and power near-real-time BI with predictable freshness.

Core Features & Use Cases

  • Primary Key upsert modeling: Use StarRocks Primary Key tables to support streaming upserts with delete-bitmaps and mutable facts/dimensions.
  • Aggregate Key & async MV pre-aggregation: Use Aggregate Key tables for immediate counter-style merges, and async materialized views for flexible dashboard aggregation refreshed on a short cadence.
  • Streaming ingestion patterns: Apply Routine Load for Kafka, Flink-to-StarRocks connector for stateful transformation, and CDC flows (Debezium/Flink CDC) for upsert/delete propagation.
  • Operational tuning: Configure batching, concurrency, checkpointing, and MV refresh intervals to meet an end-to-end latency/SLA while balancing consistency requirements.

Quick Start

Load a Kafka topic into a StarRocks Primary Key table with Routine Load for sub-minute BI, and configure max_batch_interval_s plus concurrent readers to match your freshness SLA.

Frequently Asked Questions about starrocks-realtime-modeling

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a StarRocks real-time upsert pipeline for CDC data?▼

Build a StarRocks real-time upsert pipeline by ingesting CDC event streams into Primary Key tables. Use Routine Load for Kafka topics or the Flink StarRocks connector to propagate mutable facts and dimensions with low-latency dashboard serving.

What is the best way to handle StarRocks primary key upserts from a Kafka stream?▼

StarRocks primary key upserts from Kafka are handled using Routine Load. Configure Primary Key tables with delete-bitmaps and tune max_batch_interval_s with concurrent readers to match your sub-minute freshness SLA for mutable data.

Can I use Flink CDC with StarRocks for partial column updates and deletes?▼

Yes, Flink CDC works with StarRocks for partial column updates and deletes. Use the Flink StarRocks connector to apply stateful transformations and configure appropriate upsert semantics to propagate mutable facts and dimensions correctly.

When should I use StarRocks async materialized views vs Aggregate Key tables for pre-aggregation?▼

Use StarRocks Aggregate Key tables for immediate counter-style merges during ingestion, and async materialized views for flexible dashboard aggregation. Async materialized views are refreshed on a short cadence to meet your specific freshness SLA.

How do I configure StarRocks Routine Load batching and checkpointing to meet a freshness SLA?▼

Configure StarRocks Routine Load batching by setting max_batch_interval_s and concurrent readers. Balance checkpoint and flush settings with consistency requirements to achieve end-to-end low latency while ensuring operational performance for your BI dashboards.

Does the StarRocks Flink connector support stateful transformation for real-time modeling?▼

Yes, the StarRocks Flink connector supports stateful transformation for real-time modeling. It handles streaming ingestion from CDC flows like Debezium or Flink CDC, applying upsert semantics and checkpoint configurations to maintain data consistency.

Related Skills

Explore More Software Engineering Skills→
hammadurrehman2006hammadurrehman2006

kafka-data-engineer

Design Kafka event-driven pipelines with topics, schemas, and processing logic.

Community
Advanced
ivanshamaevivanshamaev
14

starrocks-realtime-analytics

Design StarRocks real-time analytics pipelines from Kafka streams to BI-ready queries.

Community
Advanced
ivanshamaevivanshamaev
14

starrocks-cdc-pipeline

Build StarRocks CDC pipelines from PostgreSQL, MySQL, or Oracle with Debezium, Kafka, or Flink.

Community
Advanced
ivanshamaevivanshamaev
14

airflow-starrocks-pipeline

Orchestrate StarRocks ingestion from Airflow with Broker Load polling and Stream Load.

Community
Advanced