skills.rest
alexeyban
alexeybanCOMMUNITY
Explore all skills by this author →
GitHub

databricks-dq-automation

Automate post-change data quality validation for Databricks SQL outputs.

21|4|Updated Mar 9, 2026
Search Tags:#automation#validation#reporting#sql#data-quality#databricks#dq
One-click install
npx skills add https://github.com/alexeyban/databricks-lab --skill databricks-dq-automation
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-dq-automation
Source: https://github.com/alexeyban/databricks-lab/tree/main/skills/databricks-dq-automation
Command: npx skills add https://github.com/alexeyban/databricks-lab --skill databricks-dq-automation

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires databricks-sdk, python-dotenv, and includes scripts (resource) components.

What problem does it solve?

Databricks pipelines often change data structures and data flows; this skill automates the post-change validation of Databricks SQL data quality, turning pipeline updates into clear PASS/FAIL signals.

Core Features & Use Cases

  • Runs repository-defined data quality checks against Silver data using stored SQL queries in dq_queries/silver.
  • Generates a consolidated pass/fail report with per-check details and an overall status.
  • Integrates with existing pipelines and notebooks to automatically gate quality after changes, reducing manual validation effort.

Quick Start

Run the dq automation after pipeline changes to validate Silver outputs and produce a pass/fail report.

Frequently Asked Questions about databricks-dq-automation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate data quality checks in Databricks after pipeline updates?▼

Automate Databricks data quality checks by running repository-defined SQL queries against Silver layer outputs, which generates a consolidated pass/fail report to validate pipeline changes.

What is the best way to validate Databricks SQL outputs for Bronze, Silver, or Gold layers?▼

Validate Databricks SQL outputs by applying stored data quality checks from the dq_queries/silver directory after pipeline updates, producing a clear overall pass/fail status with per-check details.

Do I need a Databricks SQL endpoint to run post-change data quality validation?▼

Yes, post-change data quality validation requires a Databricks workspace with SQL endpoints configured, along with the Python dependencies databricks-sdk and python-dotenv installed in your environment.

Can I integrate automated data quality reporting into existing Databricks notebooks?▼

Yes, you can integrate automated data quality reporting into existing Databricks notebooks and pipelines to gate quality after changes, which reduces manual validation effort by generating pass/fail signals.

How does consolidated pass/fail reporting work for Databricks data quality?▼

Consolidated pass/fail reporting works by executing stored SQL checks against Silver data and aggregating the results into a single report with per-check details and an overall pass/fail status.

What are the limitations of using SQL queries for Databricks data quality automation?▼

Data quality automation is limited to validating Silver layer outputs using predefined SQL queries stored in the repository, meaning it does not dynamically generate checks or validate unsupported data layers.

Related Skills

Explore More Data & Analytics Skills→
m19cm19c

databricks-python-sdk

Manage Databricks clusters, jobs, and SQL queries via the Python SDK.

Community
Advanced
jusleejuslee
2

verify-phase

Run all phase quality gates and report pass/fail results.

Community
Advanced
RxFitRxFit

gcp-pipeline-orchestration

Generate and deploy Cloud Composer orchestration pipeline definitions for GCP workflows.

Community
Advanced
nseluganseluga

dt-qa

Write and execute tests against task 'done when:' criteria and report PASS/FAIL status.

Community
Intermediate