airflow-starrocks-cdc-orchestrator

What problem does it solve?

It solves the operational challenge of keeping StarRocks incrementally in sync with upstream CDC streams while preventing data loss during schema changes and handling load failures safely.

Core Features & Use Cases

• Watermark-based incremental sync: Computes a StarRocks watermark and loads only the delta from the source on a schedule.
• Routine Load health monitoring: Checks Kafka Routine Load state, resumes NEED_SCHEDULE jobs, and surfaces PAUSED/CANCELLED conditions.
• Flink CDC job submission & control: Starts Flink CDC jobs via REST, waits for RUNNING/terminal states, and supports cancellation for safe transitions.
• DLQ reprocessing for parse failures: Reads DLQ messages, applies fix transformations, and reloads corrected events.
• Schema change coordination: Pauses Routine Load, applies ALTER, restarts Debezium, then resumes ingestion to reduce downtime and inconsistencies.

Quick Start

Ask the agent to generate an Airflow deployment plan that wires this skill into your StarRocks CDC workflow with the required DAG schedules and placeholders for your Airflow connections (StarRocks, source DB, Kafka, and Flink REST API).