skills.rest
prashsub
prashsubCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

dlt-expectations-patterns

Enforce DLT data quality rules from a Delta table with severity filtering.

1|Updated Jan 22, 2026
Search Tags:#data quality#dlt#databricks#expectations#unity catalog#silver layer#delta table
One-click install
npx skills add https://github.com/prashsub/vibe_coding_lakehouse_starter_repo --skill dlt-expectations-patterns-prashsub
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: dlt-expectations-patterns
Source: https://github.com/prashsub/vibe_coding_lakehouse_starter_repo/tree/main/data_product_accelerator/skills/silver/01-dlt-expectations-patterns
Command: npx skills add https://github.com/prashsub/vibe_coding_lakehouse_starter_repo --skill dlt-expectations-patterns-prashsub

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) and assets (resource) components.

What problem does it solve?

This Skill addresses the challenge of managing data quality in Databricks DLT (now SDP) pipelines by providing a standardized, dynamic, and maintainable approach to defining and enforcing data quality rules.

Core Features & Use Cases

  • Delta Table for Rules: Stores data quality rules in a Unity Catalog Delta table, enabling runtime updates without code redeployment.
  • Dynamic Rule Loading: Uses a pure Python loader module to fetch rules at pipeline runtime.
  • Severity-Based Enforcement: Supports 'critical' rules (which drop or quarantine records) and 'warning' rules (which log issues but allow records to pass).
  • Quarantine Pattern: Implements a pattern to capture records failing critical checks for investigation.
  • Use Case: When building your Silver layer DLT pipelines, use this Skill to ensure data integrity by defining rules like "transaction ID must not be null" or "quantity must be positive" directly in a Delta table, which are then automatically applied by your DLT jobs.

Quick Start

Use the dlt-expectations-patterns skill to apply critical and warning data quality rules to the 'silver_transactions' table by loading them from the 'dq_rules' Delta table.

Frequently Asked Questions about dlt-expectations-patterns

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I manage DLT expectations dynamically in Databricks without redeploying pipeline code?▼

You can manage DLT expectations dynamically by storing data quality rules in a Unity Catalog Delta table. A Python loader fetches these rules at pipeline runtime, enabling updates to validation logic without code redeployment.

How does severity-based filtering work for data quality rules in a Delta table?▼

Severity-based filtering separates data quality rules into critical and warning categories. Critical rules drop or quarantine records failing validation, while warning rules log issues but allow records to pass through the pipeline.

What is the best way to quarantine failed records in a Databricks DLT Silver layer?▼

The best way to quarantine failed records is using a pattern that captures records failing critical DLT expectations. This isolates invalid data for investigation while maintaining clean silver Delta tables.

Can I use Unity Catalog to store data quality rules for Spark Declarative Pipelines?▼

Yes, you can use Unity Catalog to store data quality rules in a Delta table. This acts as a single source of truth for DLT expectations, making rules portable and maintainable across Databricks environments.

How do I load data quality expectations from a Delta table into a DLT pipeline?▼

You load data quality expectations using a pure Python loader module within your DLT pipeline. This module retrieves the validation rules from the Delta table at runtime to apply them to your silver layer transformations.

Related Skills

Explore More Data & Analytics Skills→
databricks-solutionsdatabricks-solutions
5

dlt-expectations-patterns

Enforce data quality expectations in Databricks DLT pipelines using Unity Catalog Delta table rules.

Official
Advanced
calcosmiccalcosmic
9

workflow-failure-forensics

Analyze failed colony workflows to produce forensic reports with root causes.

Community
Advanced
TheGrowthExponentTheGrowthExponent

Data Engineering Skills

Build and orchestrate Databricks ETL pipelines with Delta Live Tables.

Official
Intermediate
patrikboroschpatrikborosch
14

spark-consumption-cli

Run PySpark and Spark SQL analytics on Microsoft Fabric lakehouse data via Livy API sessions.

Community
Intermediate