observability-setup

Databricks observability setup automates the configuration of Lakehouse Monitoring, anomaly detection, AI/BI Dashboards, and SQL Alerts. It uses a manifest file to define and orchestrate data quality monitors and alerts for your Gold tables.

Schema drift detection is handled by configuring anomaly detection monitors on your Gold tables. This observability setup guides you through creating monitors that automatically identify schema-level changes and data anomalies.

Yes, end-to-end observability setup includes building AI/BI Dashboards with monitoring widgets. It orchestrates dashboard design automatically based on a config-driven manifest file to visualize data quality and operational health.

Yes, the setup leverages a config-driven manifest file to define exactly which monitors, dashboards, and SQL alerts to create. This manifest is required to orchestrate the deployment of your observability components.

You establish critical SQL alerts for data freshness and integrity through config-driven alerting. The observability setup guides you through defining these alerts in your manifest to automatically notify you of pipeline issues.

End-to-end Databricks observability includes Lakehouse Monitoring for Gold tables, schema-level anomaly detection, AI/BI Dashboards, and SQL Alerts. These components work together to ensure data quality and operational health.