airflow-starrocks-data-quality

You validate StarRocks data quality by running SQL-based checks in Airflow to assert row counts, freshness, null rates, and duplicate keys by partition. This blocks downstream consumption of invalid data by triggering a quarantine-and-drop workflow for failed partitions.

Detect StarRocks volume anomalies by applying a z-score calculation against recent historical data within an Airflow task. This statistical check identifies silent regressions in row counts post-load, allowing you to quarantine affected partitions before they impact downstream analytical models.

You quarantine bad StarRocks partitions by moving failing data to a quarantine table and dropping the original bad partition. This automated Airflow workflow ensures downstream analytical models only consume data passing referential integrity and null-rate threshold checks.

Yes, you can use Airflow to validate referential integrity between StarRocks fact and dimension tables. The data quality gate executes SQL-based checks to measure FK orphan rates, ensuring fact table records correctly reference dimension entries before allowing downstream tasks to execute.

Yes, the Airflow data quality gate supports both batch and streaming ingestion into StarRocks. It applies post-load validation tasks to verify freshness, null rates, and duplicates, ensuring row counts and volume anomalies are checked regardless of the ingestion method used.