skills.rest
ivanshamaev
ivanshamaevCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

airflow-starrocks-cdc-orchestrator

Orchestrate Kafka/Debezium and Flink CDC pipelines into StarRocks via Airflow DAGs.

14|1|Updated May 5, 2026
Search Tags:#airflow#starrocks#routine load#cdc orchestration#flink rest api#dlq reprocessing#schema change management
One-click install
npx skills add https://github.com/ivanshamaev/de-agent-skills --skill airflow-starrocks-cdc-orchestrator
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: airflow-starrocks-cdc-orchestrator
Source: https://github.com/ivanshamaev/de-agent-skills/tree/main/group_skills/starrocks_group_skills/airflow_starrocks_cdc_orchestrator
Command: npx skills add https://github.com/ivanshamaev/de-agent-skills --skill airflow-starrocks-cdc-orchestrator

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires requests, confluent-kafka.

What problem does it solve?

It solves the operational challenge of keeping StarRocks incrementally in sync with upstream CDC streams while preventing data loss during schema changes and handling load failures safely.

Core Features & Use Cases

  • Watermark-based incremental sync: Computes a StarRocks watermark and loads only the delta from the source on a schedule.
  • Routine Load health monitoring: Checks Kafka Routine Load state, resumes NEED_SCHEDULE jobs, and surfaces PAUSED/CANCELLED conditions.
  • Flink CDC job submission & control: Starts Flink CDC jobs via REST, waits for RUNNING/terminal states, and supports cancellation for safe transitions.
  • DLQ reprocessing for parse failures: Reads DLQ messages, applies fix transformations, and reloads corrected events.
  • Schema change coordination: Pauses Routine Load, applies ALTER, restarts Debezium, then resumes ingestion to reduce downtime and inconsistencies.

Quick Start

Ask the agent to generate an Airflow deployment plan that wires this skill into your StarRocks CDC workflow with the required DAG schedules and placeholders for your Airflow connections (StarRocks, source DB, Kafka, and Flink REST API).

Frequently Asked Questions about airflow-starrocks-cdc-orchestrator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I orchestrate CDC pipelines from Kafka into StarRocks using Airflow?▼

You can orchestrate CDC pipelines into StarRocks using Airflow DAGs that compute watermarks, load deltas, and monitor Routine Load states. The skill coordinates Flink job submissions via REST and handles DLQ replay to prevent data loss.

What is the best way to handle StarRocks Routine Load failures and lag?▼

Handling StarRocks Routine Load failures involves checking the job state via SHOW commands, automatically resuming NEED_SCHEDULE jobs, and surfacing PAUSED or CANCELLED conditions. The skill monitors these states to maintain healthy ingestion.

How do you coordinate schema changes in StarRocks without losing CDC data?▼

Coordinating schema changes requires pausing Routine Load, applying the ALTER operation, restarting Debezium, and then resuming ingestion. This sequence minimizes downtime and prevents data inconsistencies during the schema rollout.

Can I reprocess Kafka dead letter queue messages in a StarRocks CDC pipeline?▼

Yes, you can reprocess DLQ messages by consuming them from Kafka, applying fix transformations to correct parse failures, and reloading the events into StarRocks. This ensures failed records are recovered safely.

How do I submit and monitor Flink CDC jobs to StarRocks via REST API?▼

You can submit and monitor Flink CDC jobs by sending requests to the Flink REST API, polling for RUNNING or terminal states, and supporting job cancellation. This ensures safe transitions during pipeline updates.

Do I need Airflow connections configured before running StarRocks CDC guardrails?▼

Yes, you need placeholders for Airflow connections covering StarRocks, the source database, Kafka, and the Flink REST API. These connections are required to execute the DAG schedules and health guards properly.

Related Skills

Explore More Software Engineering Skills→
ivanshamaevivanshamaev
14

starrocks-routine-load-kafka

Configure StarRocks Routine Load for Kafka ingestion with JSON, CSV, and Avro formats.

Community
Advanced
ivanshamaevivanshamaev
14

starrocks-cdc-pipeline

Build StarRocks CDC pipelines from PostgreSQL, MySQL, or Oracle with Debezium, Kafka, or Flink.

Community
Advanced
hammadurrehman2006hammadurrehman2006

kafka-data-engineer

Design Kafka event-driven pipelines with topics, schemas, and processing logic.

Community
Advanced
ivanshamaevivanshamaev
14

airflow-starrocks-pipeline

Orchestrate StarRocks ingestion from Airflow with Broker Load polling and Stream Load.

Community
Advanced