skills.rest
databricks-solutions
databricks-solutionsOFFICIAL
Explore all skills by this author →
GitHub

dlt-expectations-patterns

Enforce data quality expectations in Databricks DLT pipelines using Unity Catalog Delta table rules.

5|6|Updated Jan 26, 2026
Search Tags:#validation#data quality#pipeline#dlt#databricks#expectations#delta lake
One-click install
npx skills add https://github.com/databricks-solutions/vibe-coding-workshop-template --skill dlt-expectations-patterns
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: dlt-expectations-patterns
Source: https://github.com/databricks-solutions/vibe-coding-workshop-template/tree/main/data_product_accelerator/skills/silver/01-dlt-expectations-patterns
Command: npx skills add https://github.com/databricks-solutions/vibe-coding-workshop-template --skill dlt-expectations-patterns

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) and assets (resource) components.

What problem does it solve?

This Skill addresses the challenge of maintaining robust data quality in Databricks DLT pipelines by providing a standardized, dynamic, and manageable approach to defining and enforcing data quality expectations.

Core Features & Use Cases

  • Dynamic Expectations: Load data quality rules from a Unity Catalog Delta table, allowing updates without redeploying pipeline code.
  • Severity Levels: Differentiate between critical rules (which can cause data to be dropped or quarantined) and warning rules (which are logged but allow data to pass).
  • Quarantine Pattern: Automatically capture records that fail critical checks into a dedicated quarantine table for investigation.
  • Use Case: Implement a Silver layer DLT pipeline where critical fields like transaction_id and store_number must be present, and reasonable ranges are enforced for quantity_sold. If a transaction is missing an ID, it's quarantined with a specific reason.

Quick Start

Use the dlt-expectations-patterns skill to implement data quality checks for your Silver layer DLT pipeline by defining rules in a Delta table and loading them dynamically.

Frequently Asked Questions about dlt-expectations-patterns

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I manage data quality expectations in Databricks DLT without redeploying pipeline code?▼

You can manage data quality expectations dynamically by loading validation rules from a Unity Catalog Delta table. This approach allows you to update data quality checks at runtime without redeploying your DLT pipeline code.

What is the best way to quarantine failed records in a Delta Live Tables pipeline?▼

The best way to quarantine failed records in a DLT pipeline is by using expectations with critical severity. This pattern automatically captures records that fail critical checks into a dedicated quarantine table for investigation.

How does DLT handle different severity levels for data quality validation?▼

DLT handles data quality validation by differentiating between critical and warning severities. Critical rules can drop or quarantine records, while warning rules log issues but allow the data to pass through the pipeline.

Can I load data quality rules from a Delta table into Databricks DLT using Python?▼

Yes, you can load data quality rules from a Delta table into Databricks DLT using pure Python modules. This method avoids DLT static analysis warnings while enabling the dynamic runtime loading of expectations.

Why does DLT static analysis warn when defining expectations directly in pipeline code?▼

DLT static analysis warns when expectations are defined directly because it struggles to track dynamically generated rules. Using pure Python modules to load rules from an external table bypasses these warnings and enables runtime updates.

Related Skills

Explore More Data & Analytics Skills→
d-padmanabhand-padmanabhan
15

databricks

Design and standardize Databricks governance workflows for workspace bootstrap, Unity Catalog migration, and DLT pipelines.

Community
Advanced
dadbodgeoffdadbodgeoff
783

validation-quarantine

Validate incoming data against a Zod schema and quality scoring function.

Community
Advanced
vasilyu1983vasilyu1983
73

data-lake-platform

Automate data lake and lakehouse design and deployment across ingestion, transformation, storage, and orchestration.

Community
Advanced
prashsubprashsub
1

dlt-expectations-patterns

Enforce DLT data quality rules from a Delta table with severity filtering.

Community
Advanced