starrocks-data-quality-guardian

What problem does it solve?

StarRocks data quality problems often surface late—after downstream jobs fail or dashboards lie—so freshness, duplicates, null anomalies, volume drift, integrity, and storage-health issues need consistent, lightweight detection.

Core Features & Use Cases

• Freshness monitoring: Detect stale data via MAX(updated_at) against an SLA, including per-partition freshness checks.
• Anomaly detection: Surface duplicate keys in staging tables, measure column-level null-rate anomalies against historical baselines, and detect day-over-day volume drift.
• Integrity & completeness validation: Check referential integrity (orphan fact rows), cross-table row-count reconciliation, and missing partitions/completeness coverage.
• StarRocks-specific health checks: Validate tablet replication/health signals, compaction backlog indicators, and routine load error states.
• Operational DQ scanning: Run the checks via a Python DQ scan class and store results for trend monitoring.

Quick Start

Ask the agent to validate yesterday’s StarRocks data in sales.orders by running freshness, duplicate, null-rate, volume drift, and partition completeness checks and then saving the pass/fail metrics into monitoring.dq_results.