skills.rest
prashsub
prashsubCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

02-merge-patterns

Implements PySpark MERGE patterns for Silver-to-Gold data with SCD Type 1/2.

1|Updated Jan 22, 2026
Search Tags:#data modeling#merge#pyspark#scd type 2#delta lake#gold layer#scd type 1
One-click install
npx skills add https://github.com/prashsub/vibe_coding_lakehouse_starter_repo --skill 02-merge-patterns-prashsub
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: 02-merge-patterns
Source: https://github.com/prashsub/vibe_coding_lakehouse_starter_repo/tree/main/data_product_accelerator/skills/gold/pipeline-workers/02-merge-patterns
Command: npx skills add https://github.com/prashsub/vibe_coding_lakehouse_starter_repo --skill 02-merge-patterns-prashsub

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes assets (resource) components.

What problem does it solve?

This Skill provides robust, production-grade patterns for performing MERGE operations when moving data from the Silver to the Gold layer in a data lakehouse, ensuring data integrity and handling complex transformations.

Core Features & Use Cases

  • Schema Evolution Handling: Manages changes in column names and data types between Silver and Gold tables.
  • SCD Type 1 & 2 Implementation: Supports both overwrite (Type 1) and historical tracking (Type 2) for dimension tables.
  • Fact Table Aggregation: Provides patterns for aggregating transactional data into fact tables.
  • Use Case: When migrating customer data from a Silver table with cust_id to a Gold dimension table expecting customer_identifier, this Skill ensures the column mapping is handled correctly, and if it's an SCD Type 2 dimension, it properly updates the effective_to and is_current flags for the old record and inserts a new one.

Quick Start

Use the 02-merge-patterns skill to implement a MERGE operation for the 'dim_customer' table, handling SCD Type 2 logic.

Frequently Asked Questions about 02-merge-patterns

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I implement SCD Type 2 in a Delta Lake Gold layer using PySpark?▼

Implement SCD Type 2 in a Delta Lake Gold layer by using MERGE operations to update effective_to and is_current flags for old records while inserting new ones for historical tracking.

What is the best way to handle column mapping differences between Silver and Gold tables in PySpark?▼

Handle column mapping differences between Silver and Gold tables by applying production-grade MERGE patterns that explicitly map mismatched columns like cust_id to customer_identifier during the data transformation.

How do I aggregate fact tables when moving data from Silver to Gold in Delta Lake?▼

Aggregate fact tables when moving data from Silver to Gold in Delta Lake by using MERGE patterns designed for aggregating transactional data before merging it into the target Gold layer.

Why does my PySpark MERGE operation fail due to variable naming conflicts?▼

PySpark MERGE operations fail due to variable naming conflicts when column names clash with built-in PySpark functions, which can be prevented by using production-grade naming conventions in your MERGE patterns.

Does this Gold layer MERGE pattern support SCD Type 1 for dimension tables?▼

Yes, the Gold layer MERGE pattern supports SCD Type 1 for dimension tables by directly overwriting existing records with updated values without retaining historical data.

How do I manage schema evolution when merging Silver data into a Gold layer?▼

Manage schema evolution when merging Silver data into a Gold layer by using MERGE patterns that account for changes in column names and data types between the source and target tables.

Related Skills

Explore More Data & Analytics Skills→
databricks-solutionsdatabricks-solutions
5

02-merge-patterns

Provides Gold layer MERGE patterns including SCD Type 1/2 and schema evolution for PySpark data lakes.

Official
Advanced
prashsubprashsub
1

gold-layer-design

Orchestrate Gold layer design with dimensional modeling and YAML schema generation.

Community
Advanced
erichowenserichowens
181

data-pipeline-engineer

Automate ETL/ELT pipeline design, validation, and operation across medallion architectures.

Community
Advanced
Gami19Gami19

delta-lake

Design, optimize, and operate Delta Lake tables across Bronze, Silver, and Gold layers.

Community
Advanced