dqx-patterns

Advanced data quality validation in Databricks Silver and Gold layer pipelines is implemented using the DQX framework, YAML-based check definitions, and flexible quarantine strategies that split invalid records from valid data.

DQX quarantine captures detailed failure diagnostics and applies configurable strategies like drop, mark, or split to invalid data, offering more granular control than standard DLT expectations for handling records failing checks.

Yes, installing the DQX library is required to run data quality checks. You apply checks defined in a YAML file to your input DataFrame and split the results into valid and invalid DataFrames.

The best way to capture detailed failure reasons for data quality checks in Databricks is using the DQX framework, which automatically identifies and records specific reasons for null critical fields or values outside acceptable ranges.

Yes, DQX checks integrate with DLT and Lakeflow pipelines to enforce Silver layer data quality, automatically identifying invalid records and applying quarantine strategies based on your YAML definitions.