anomaly-detection

Automate data quality monitoring for Databricks Unity Catalog tables by enabling schema-level anomaly detection to check table freshness and completeness using ML models without manual metric definition.

Anomaly detection for data freshness and completeness in Databricks uses ML models to automatically identify stale or incomplete tables, alerting you to potential data pipeline breaks before downstream impact.

Yes, you can monitor an entire schema for data quality issues in Unity Catalog by applying schema-level anomaly detection across all tables within that specific catalog schema.

Set up automated data reliability checks for gold layer tables by integrating anomaly detection into your schema setup workflows to continuously monitor data volume expectations and table freshness.

Databricks data quality monitoring results are stored in the system.data_quality_monitoring.table_results location, providing a centralized repository for tracking table freshness and completeness metrics over time.

No, you do not need to manually define metrics for Databricks schema monitoring because this approach uses ML models to automatically detect anomalies in table freshness and completeness.