skills.rest
ivanshamaev
ivanshamaevCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

starrocks-routine-load-kafka

Configure StarRocks Routine Load for Kafka ingestion with JSON, CSV, and Avro formats.

14|1|Updated May 5, 2026
Search Tags:#kafka#cdc#starrocks#exactly-once#routine load#jsonpaths#sql ddl
One-click install
npx skills add https://github.com/ivanshamaev/de-agent-skills --skill starrocks-routine-load-kafka
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: starrocks-routine-load-kafka
Source: https://github.com/ivanshamaev/de-agent-skills/tree/main/group_skills/starrocks_group_skills/starrocks_routine_load_kafka
Command: npx skills add https://github.com/ivanshamaev/de-agent-skills --skill starrocks-routine-load-kafka

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill solves the problem of setting up continuous, low-latency ingestion from Kafka into StarRocks using Routine Load, including correct configuration for formats, concurrency, and operational control.

Core Features & Use Cases

  • Kafka → StarRocks Routine Load DDL generation: produces a complete CREATE ROUTINE LOAD statement with LOAD PROPERTIES and FROM KAFKA clauses.
  • Multi-format parsing for ingestion: configures JSON/CSV/Avro parsing (including JSONPaths and Avro Schema Registry URL).
  • Operational controls and production tuning: covers desired_concurrent_number tuning, PAUSE/RESUME/ALTER/STOP workflows, SHOW ROUTINE LOAD inspection, and error-log driven troubleshooting (including offset resets after Kafka retention).
  • CDC upsert pattern for Primary Key tables: supports exactly-once style semantics via offset commit after tablet commit and provides a practical upsert mapping approach with CDC envelope fields and filtering.
  • Monitoring guidance: explains how to calculate consumer lag by correlating Kafka high-watermarks with StarRocks committed progress.

Quick Start

Create a StarRocks routine load job from a Kafka CDC topic by defining all PROPERTIES and KAFKA parameters in a single CREATE ROUTINE LOAD statement for your target table.

Frequently Asked Questions about starrocks-routine-load-kafka

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I configure StarRocks Routine Load to ingest data from a Kafka topic?▼

Configure StarRocks Routine Load to ingest Kafka data by defining a complete CREATE ROUTINE LOAD statement. This includes specifying LOAD PROPERTIES for formats like JSON or CSV, and a FROM KAFKA clause with brokers, topic, and partition details for continuous upserts.

Does StarRocks Routine Load support exactly-once semantics for CDC upserts?▼

Yes, StarRocks Routine Load supports exactly-once style semantics for CDC upserts into Primary Key tables. This is achieved by committing Kafka offsets only after tablet commits succeed, ensuring reliable streaming data ingestion without duplication.

How do I monitor consumer lag and troubleshoot errors in StarRocks Routine Load?▼

Monitor consumer lag by correlating Kafka high-watermarks with StarRocks committed progress using SHOW ROUTINE LOAD. Error diagnosis is driven by error logs, supporting operational workflows like PAUSE, RESUME, ALTER, and STOP to manage ingestion issues.

Can I use Avro Schema Registry with StarRocks Routine Load for Kafka ingestion?▼

Yes, you can use Avro Schema Registry with StarRocks Routine Load. The configuration supports multi-format parsing including JSON, CSV, and Avro, allowing you to specify the Avro Schema Registry URL for continuous low-latency data ingestion.

What's the best way to handle Kafka offset resets after retention periods in StarRocks?▼

Handle Kafka offset resets after retention edge cases in StarRocks by using the ALTER ROUTINE LOAD workflow. This operational control allows you to reset offsets and resume continuous ingestion when historical Kafka data is no longer available.

How do I tune StarRocks Routine Load concurrency and batch sizes for streaming analytics?▼

Tune StarRocks Routine Load concurrency and batch sizes using the desired_concurrent_number property and batch/error tuning parameters within the CREATE ROUTINE LOAD statement. This production tuning optimizes streaming analytics workloads for low-latency ingestion.

Related Skills

Explore More Data & Analytics Skills→
ivanshamaevivanshamaev
14

starrocks-realtime-modeling

Designs StarRocks ingestion and modeling for mutable tables via Kafka, Flink, or CDC streams.

Community
Advanced
m19cm19c

databricks-spark-structured-streaming

Build and optimize Spark Structured Streaming pipelines with Kafka integration.

Community
Advanced
ivanshamaevivanshamaev
14

starrocks-cdc-pipeline

Build StarRocks CDC pipelines from PostgreSQL, MySQL, or Oracle with Debezium, Kafka, or Flink.

Community
Advanced
ivanshamaevivanshamaev
14

airflow-starrocks-pipeline

Orchestrate StarRocks ingestion from Airflow with Broker Load polling and Stream Load.

Community
Advanced