airflow-starrocks-etl-best-practices

Design idempotent Airflow ETL pipelines loading data into StarRocks.

14|1|Updated May 5, 2026

Search Tags:#etl #airflow #idempotency #starrocks #broker load #partition overwrite #sla monitoring

Ready to run

npx skills add https://github.com/ivanshamaev/de-agent-skills --skill airflow-starrocks-etl-best-practices

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: airflow-starrocks-etl-best-practices
Source: https://github.com/ivanshamaev/de-agent-skills/tree/main/group_skills/starrocks_group_skills/airflow_starrocks_etl_best_practices
Command: npx skills add https://github.com/ivanshamaev/de-agent-skills --skill airflow-starrocks-etl-best-practices

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill prevents duplicate or corrupted data when you repeatedly run Airflow ETL jobs that load into StarRocks, especially during retries and backfills.

Core Features & Use Cases

Idempotent DAG loading patterns: partition replacement via INSERT OVERWRITE, deterministic Broker Load labels, and retry-safe execution.
Deduplication and upsert handling: stable dedup logic before insert and safe merge/upsert behavior for primary-key tables.
Production reliability and observability: task retries with exponential backoff, SLA miss callbacks, freshness checks, and lineage tagging for audit trails.
Operational safety for backfills: dependency ordering, concurrency controls, catchup safety checklist, and dynamic partition management with duplicate prevention.

Quick Start

Ask the agent to generate an Airflow DAG for StarRocks that implements deterministic Broker Load labels, uses INSERT OVERWRITE for partition replacement, adds retries with exponential backoff, includes an SLA miss callback, and performs freshness validation after each run.

Frequently Asked Questions about airflow-starrocks-etl-best-practices

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I prevent duplicate data in StarRocks when Airflow ETL jobs retry or backfill?▼

To prevent duplicate data in StarRocks during Airflow retries or backfills, implement idempotent pipelines using deterministic Broker Load labels and INSERT OVERWRITE for partition replacement. This ensures repeated runs safely replace target partitions without creating duplicate rows or race conditions.

What is the best way to configure idempotent Airflow DAGs for StarRocks broker loads?▼

The best way to configure idempotent Airflow DAGs for StarRocks is to use deterministic Broker Load labels combined with partition-based ingestion strategies. Implementing INSERT OVERWRITE allows the pipeline to replace specific partitions cleanly, ensuring retry safety and preventing duplicate data loads.

How does INSERT OVERWRITE handle partition replacement in StarRocks ETL pipelines?▼

INSERT OVERWRITE handles partition replacement in StarRocks by atomically overwriting data in a target partition during the ETL pipeline run. This mechanism replaces existing records with the new dataset, ensuring that Airflow task retries do not append duplicate data to previously loaded partitions.

Can I use Airflow SLA monitoring and freshness validation for StarRocks daily loads?▼

Yes, you can use Airflow SLA monitoring and freshness validation for StarRocks daily loads by configuring SLA miss callbacks and performing freshness checks after each run. Adding task retries with exponential backoff and lineage tagging further ensures production reliability and auditability.

How do I manage dynamic partition creation and concurrency limits in Airflow StarRocks pipelines?▼

To manage dynamic partition creation and concurrency limits in Airflow StarRocks pipelines, apply dependency ordering and concurrency controls to prevent race conditions. Using dynamic partition management with duplicate prevention ensures safe catchup execution during complex multi-table dependency backfills.