airflow-starrocks-data-quality

Automate post-load data quality gating for StarRocks tables in Airflow.

14|1|Updated May 5, 2026

Search Tags:#data quality #airflow #starrocks #freshness checks #dq gates #partition quarantine #z-score anomaly detection

Ready to run

npx skills add https://github.com/ivanshamaev/de-agent-skills --skill airflow-starrocks-data-quality

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: airflow-starrocks-data-quality
Source: https://github.com/ivanshamaev/de-agent-skills/tree/main/group_skills/starrocks_group_skills/airflow_starrocks_data_quality
Command: npx skills add https://github.com/ivanshamaev/de-agent-skills --skill airflow-starrocks-data-quality

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps prevent broken or incomplete StarRocks data from reaching downstream pipelines by adding automated data quality gates directly after loads.

Core Features & Use Cases

Post-load DQ validation: Verifies row counts, data freshness, null-rate thresholds, and duplicate-key violations after Broker Load or Stream Load.
Anomaly and integrity checks: Detects volume anomalies using a z-score against recent history and validates referential integrity between fact and dimension tables.
Quarantine bad partitions: Moves failing partition data to a quarantine table and drops the bad partition to protect downstream analytical models.

Quick Start

Use the Skill by running its DQ Gate DAG in Airflow to validate an orders partition (row count, freshness, null rates, duplicates, volume anomaly, and FK orphan rate) before allowing downstream tasks to execute.

Frequently Asked Questions about airflow-starrocks-data-quality

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I validate data quality in StarRocks after a batch or streaming load?▼

You validate StarRocks data quality by running SQL-based checks in Airflow to assert row counts, freshness, null rates, and duplicate keys by partition. This blocks downstream consumption of invalid data by triggering a quarantine-and-drop workflow for failed partitions.

What is the best way to detect volume anomalies in StarRocks partition data?▼

Detect StarRocks volume anomalies by applying a z-score calculation against recent historical data within an Airflow task. This statistical check identifies silent regressions in row counts post-load, allowing you to quarantine affected partitions before they impact downstream analytical models.

How do I quarantine bad partitions in StarRocks to protect downstream pipelines?▼

You quarantine bad StarRocks partitions by moving failing data to a quarantine table and dropping the original bad partition. This automated Airflow workflow ensures downstream analytical models only consume data passing referential integrity and null-rate threshold checks.

Can I use Airflow to check referential integrity between StarRocks fact and dimension tables?▼

Yes, you can use Airflow to validate referential integrity between StarRocks fact and dimension tables. The data quality gate executes SQL-based checks to measure FK orphan rates, ensuring fact table records correctly reference dimension entries before allowing downstream tasks to execute.

Does this Airflow data quality gate support both batch and streaming ingestion into StarRocks?▼

Yes, the Airflow data quality gate supports both batch and streaming ingestion into StarRocks. It applies post-load validation tasks to verify freshness, null rates, and duplicates, ensuring row counts and volume anomalies are checked regardless of the ingestion method used.