starrocks-routine-load-kafka

What problem does it solve?

This Skill solves the problem of setting up continuous, low-latency ingestion from Kafka into StarRocks using Routine Load, including correct configuration for formats, concurrency, and operational control.

Core Features & Use Cases

• Kafka → StarRocks Routine Load DDL generation: produces a complete CREATE ROUTINE LOAD statement with LOAD PROPERTIES and FROM KAFKA clauses.
• Multi-format parsing for ingestion: configures JSON/CSV/Avro parsing (including JSONPaths and Avro Schema Registry URL).
• Operational controls and production tuning: covers desired_concurrent_number tuning, PAUSE/RESUME/ALTER/STOP workflows, SHOW ROUTINE LOAD inspection, and error-log driven troubleshooting (including offset resets after Kafka retention).
• CDC upsert pattern for Primary Key tables: supports exactly-once style semantics via offset commit after tablet commit and provides a practical upsert mapping approach with CDC envelope fields and filtering.
• Monitoring guidance: explains how to calculate consumer lag by correlating Kafka high-watermarks with StarRocks committed progress.

Quick Start

Create a StarRocks routine load job from a Kafka CDC topic by defining all PROPERTIES and KAFKA parameters in a single CREATE ROUTINE LOAD statement for your target table.