skills.rest
ivanshamaev
ivanshamaevCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

airflow-starrocks-pipeline

Orchestrate StarRocks ingestion from Airflow with Broker Load polling and Stream Load.

14|1|Updated May 5, 2026
Search Tags:#orchestration#airflow#starrocks#broker load#routine load#stream load#etl pipelines
One-click install
npx skills add https://github.com/ivanshamaev/de-agent-skills --skill airflow-starrocks-pipeline
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: airflow-starrocks-pipeline
Source: https://github.com/ivanshamaev/de-agent-skills/tree/main/group_skills/starrocks_group_skills/airflow_starrocks_pipeline
Command: npx skills add https://github.com/ivanshamaev/de-agent-skills --skill airflow-starrocks-pipeline

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

It helps you reliably orchestrate StarRocks ingestion workflows from Airflow, including async Broker Load polling and Routine Load lifecycle management, so ETL pipelines don’t get stuck or load with stale metadata.

Core Features & Use Cases

  • Broker Load orchestration (S3 batch ingestion): Create partitions, trigger Broker Load jobs with a deterministic label, and poll SHOW LOAD ... until FINISHED.
  • Stream Load micro-batch pattern: Send NDJSON payloads via HTTP with a computed label and validate StarRocks response status.
  • Routine Load lifecycle control: Pause for schema changes, apply DDL, and resume while handling job state transitions safely.
  • Post-load optimization: Run ANALYZE TABLE for partitions to refresh statistics and improve query planning.
  • Airflow integration details: Use MySqlHook for MySQL-compatible DDL/DML and Http/requests for Stream Load; pass labels via XCom and template partition-aware DAG parameters.

Quick Start

Ask the agent to generate an Airflow DAG that performs partition-aware Broker Load into a StarRocks table, polls until completion, then runs ANALYZE TABLE ... PARTITION (...) and basic row-count validation for the same date.

Frequently Asked Questions about airflow-starrocks-pipeline

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I orchestrate StarRocks Broker Load jobs in Airflow and poll until completion?▼

Orchestrate StarRocks Broker Load in Airflow by triggering jobs with deterministic labels and polling SHOW LOAD until FINISHED. This prevents ETL pipelines from getting stuck during S3 batch ingestion.

How do I submit micro-batch Stream Load payloads to StarRocks from an Airflow DAG?▼

Submit Stream Load payloads by sending NDJSON via HTTP with computed labels and JSONPaths. Validate the StarRocks response status to ensure near-real-time micro-batch loads succeed.

How do I safely handle StarRocks Routine Load during schema changes in Airflow?▼

Handle Routine Load safely by pausing jobs for schema changes, applying DDL, then resuming. This lifecycle control ensures ongoing ingestion jobs transition states without breaking.

Do I need a MySQL connection to run DDL and load state queries against StarRocks from Airflow?▼

You need a MySQL-compatible StarRocks FE connection to run DDL and load state queries via Airflow's MySqlHook. This allows you to manage partitions and monitor ingestion job status.

What is the best way to refresh statistics after a StarRocks partition-aware batch load in Airflow?▼

Refresh statistics after batch loads by running ANALYZE TABLE for specific partitions with proper async modes. This improves query planning and validates data with row-count checks.

Why does my StarRocks ingestion pipeline load with stale metadata when orchestrated through Airflow?▼

StarRocks ingestion pipelines load with stale metadata without post-load ANALYZE TABLE execution. Running partition-aware statistics refreshes metadata and prevents poor query planning.

Related Skills

Explore More Software Engineering Skills→
StarRocksStarRocks
73

import

Diagnose StarRocks import slowness, timeouts, and RPC failures.

Official
Advanced
ivanshamaevivanshamaev
14

starrocks-cbo

Correct StarRocks CBO statistics and join strategies using ANALYZE TABLE and EXPLAIN COSTS.

Community
Advanced
ivanshamaevivanshamaev
14

starrocks-broker-load

Automate bulk data loading into StarRocks via Broker Load from cloud storage.

Community
Intermediate
k1lgork1lgor
2

workflow-orchestrator

Orchestrate scheduled and event-driven workflows across Airflow, Celery, and Cron.

Community
Advanced