skills.rest
databricks-solutions
databricks-solutionsOFFICIAL
Explore all skills by this author →
GitHub

02-merge-patterns

Provides Gold layer MERGE patterns including SCD Type 1/2 and schema evolution for PySpark data lakes.

5|6|Updated Jan 26, 2026
Search Tags:#data modeling#merge#etl#pyspark#databricks#delta lake#gold layer
One-click install
npx skills add https://github.com/databricks-solutions/vibe-coding-workshop-template --skill 02-merge-patterns
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: 02-merge-patterns
Source: https://github.com/databricks-solutions/vibe-coding-workshop-template/tree/main/data_product_accelerator/skills/gold/pipeline-workers/02-merge-patterns
Command: npx skills add https://github.com/databricks-solutions/vibe-coding-workshop-template --skill 02-merge-patterns

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) and assets (resource) components.

What problem does it solve?

This Skill provides robust and production-grade patterns for performing MERGE operations when transitioning data from the Silver to the Gold layer in a data lakehouse, ensuring data integrity and consistency.

Core Features & Use Cases

  • SCD Type 1 & 2: Implements patterns for handling slowly changing dimensions, allowing for either overwriting historical data or tracking changes over time.
  • Fact Table Aggregation: Offers templates for aggregating transactional data from Silver into summarized fact tables in the Gold layer.
  • Schema Evolution: Provides guidance on handling data type changes and adding derived columns during the merge process.
  • Use Case: When migrating customer data from a Silver table to a Gold dimension table, use this Skill to correctly apply SCD Type 2 logic, ensuring that historical changes in customer addresses or status are preserved and queryable.

Quick Start

Use the 02-merge-patterns skill to implement an SCD Type 1 merge for the dim_product table.

Frequently Asked Questions about 02-merge-patterns

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I implement SCD Type 2 merge operations in Delta Lake for a Gold dimension table?▼

SCD Type 2 merge operations in Delta Lake track historical changes by preserving previous records and inserting updated ones. This Skill provides production-grade PySpark patterns to correctly apply SCD Type 2 logic, ensuring historical changes in dimension attributes are queryable.

What is the best way to handle schema evolution and column mapping during a Silver to Gold merge?▼

Handling schema evolution during a Silver to Gold merge requires mapping differing column names and managing data type changes. This Skill provides templates to add derived columns and correctly map schemas, preventing variable naming conflicts with PySpark functions during the merge process.

How do I aggregate fact tables from Silver to Gold using PySpark?▼

Aggregating fact tables from Silver to Gold involves summarizing transactional data into structured outputs. This Skill offers specific PySpark merge templates to aggregate transactional records efficiently, maintaining data integrity and consistency within the lakehouse architecture.

Why does my Gold layer MERGE fail due to variable naming conflicts with PySpark functions?▼

Gold layer MERGE operations can fail when column names conflict with PySpark function names during schema mapping. This Skill prevents these variable naming conflicts by providing robust merge patterns that correctly alias and reference columns throughout the Delta Lake operation.

Does this Skill provide patterns for both SCD Type 1 and SCD Type 2 upserts in Databricks?▼

This Skill provides production-grade patterns for both SCD Type 1 and SCD Type 2 upserts in Databricks. SCD Type 1 overwrites historical data while SCD Type 2 tracks changes over time, allowing you to choose the appropriate dimension modeling approach for your Gold layer.

When do I need to use MERGE operations instead of overwrite for Delta Lake Gold tables?▼

MERGE operations are needed for Delta Lake Gold tables when performing upserts to preserve existing records and apply incremental updates. Use this Skill when transitioning data from Silver to Gold to ensure data integrity, handle schema evolution, and implement fact table aggregations.

Related Skills

Explore More Data & Analytics Skills→
prashsubprashsub
1

02-merge-patterns

Implements PySpark MERGE patterns for Silver-to-Gold data with SCD Type 1/2.

Community
Advanced
m19cm19c

databricks-aibi-dashboards

Automate creation, testing, and deployment of Databricks AI/BI dashboards.

Community
Advanced
erichowenserichowens
181

data-pipeline-engineer

Automate ETL/ELT pipeline design, validation, and operation across medallion architectures.

Community
Advanced
ivanshamaevivanshamaev
14

pyspark-structured-streaming

Build PySpark Structured Streaming pipelines for Kafka, file, and rate sources.

Community
Advanced