starrocks-realtime-analytics

What problem does it solve?

This Skill solves the problem of delivering sub-minute analytics from Kafka streams while keeping BI queries low-latency and dashboard-friendly in StarRocks.

Core Features & Use Cases

• Kafka → Routine Load ingestion: set up near-real-time ingestion with tuned batch interval, concurrency, JSON mapping, strict mode, and UTC handling.
• Primary Key real-time tables: design StarRocks Primary Key tables with range partitioning and low-latency storage settings for upsert-like updates.
• Real-time serving for dashboards: write time-window metric queries and use async materialized views for repeated aggregations to reduce query cost.
• Low-latency multi-table querying: apply colocate join patterns so join execution avoids unnecessary shuffle during real-time queries.
• Operational stability: isolate ingestion vs OLAP workload using resource groups to prevent contention.
• Freshness monitoring: check ingestion lag (seconds behind latest record) to drive alerts for real-time dashboards.

Use Case example: you ingest order events from Kafka and power a live “revenue last 15 minutes” dashboard with pre-aggregated MV-backed queries refreshed every minute.

Quick Start

Create a StarRocks Primary Key table for your events, configure Routine Load from your Kafka topic, and query the last-15-minute metrics in Grafana using minute-level time aggregation.