skills.rest
databricks-solutions
databricks-solutionsOFFICIAL
Explore all skills by this author →
GitHub

05-schema-validation

Validate DataFrame schemas against Gold layer Delta Lake table DDLs.

5|6|Updated Jan 26, 2026
Search Tags:#data integrity#data quality#schema validation#delta lake#gold layer#merge operation
One-click install
npx skills add https://github.com/databricks-solutions/vibe-coding-workshop-template --skill 05-schema-validation
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: 05-schema-validation
Source: https://github.com/databricks-solutions/vibe-coding-workshop-template/tree/main/data_product_accelerator/skills/gold/pipeline-workers/05-schema-validation
Command: npx skills add https://github.com/databricks-solutions/vibe-coding-workshop-template --skill 05-schema-validation

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill prevents common Gold layer bugs caused by schema mismatches between DataFrames and target Delta tables, ensuring data quality and preventing runtime errors during merge operations.

Core Features & Use Cases

  • Runtime Schema Validation: Compares DataFrame columns and data types against the actual DDL schema of the target Gold table before a MERGE operation.
  • Error Prevention: Catches UNRESOLVED_COLUMN errors and other schema-related issues before they impact production.
  • Use Case: When preparing a DataFrame for a dim_user table merge, this Skill will verify that all required columns from the dim_user DDL are present in the DataFrame and that no extra columns are included, preventing merge failures.

Quick Start

Validate the schema of the updates_df DataFrame against the catalog.schema.dim_user table before performing a merge.

Frequently Asked Questions about 05-schema-validation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I prevent UNRESOLVED_COLUMN errors during Delta Lake merge operations?▼

To prevent UNRESOLVED_COLUMN errors during Delta Lake merge operations, validate runtime schema consistency between DataFrames and target Gold layer tables using Spark SQL before executing the merge.

What is runtime schema validation for Gold layer Delta tables?▼

Runtime schema validation for Gold layer Delta tables is the process of comparing DataFrame columns and data types against the target table's actual DDL schema to enforce data integrity before merge operations.

How do I validate a DataFrame schema against a Delta table DDL before merging?▼

Validate a DataFrame schema against a Delta table DDL by using helper functions to compare columns and data types, ensuring the DDL acts as the source of truth for accurate column mapping and type compatibility.

Why does my Spark SQL merge fail with schema mismatches in the Gold layer?▼

Spark SQL merges fail with schema mismatches in the Gold layer when DataFrame columns do not align with the target Delta table DDL, causing unresolved columns or type incompatibilities during the operation.

Can I check for extra or missing columns in a DataFrame before a Delta Lake merge?▼

Yes, you can check for extra or missing columns in a DataFrame before a Delta Lake merge by running pre-deployment validation scripts that verify all required DDL columns are present and no extras exist.

Related Skills

Explore More Data & Analytics Skills→
databricks-solutionsdatabricks-solutions
5

gold-layer-setup

Create Databricks Gold layer tables from YAML schema definitions.

Official
Intermediate
ThomazRossitoThomazRossito
31

Microsoft Fabric Lakehouse — Medallion Architecture

Design and implement a Medallion Lakehouse pattern in Microsoft Fabric.

Community
Advanced
erichowenserichowens
181

data-pipeline-engineer

Automate ETL/ELT pipeline design, validation, and operation across medallion architectures.

Community
Advanced
databricks-solutionsdatabricks-solutions
5

02-merge-patterns

Provides Gold layer MERGE patterns including SCD Type 1/2 and schema evolution for PySpark data lakes.

Official
Advanced