skills.rest
databricks-solutions
databricks-solutionsOFFICIAL
Explore all skills by this author →
GitHub

04-grain-validation

Validate DataFrame grain against Gold layer DDL PRIMARY KEY definitions before merge operations.

5|6|Updated Jan 26, 2026
Search Tags:#data integrity#data warehousing#merge validation#gold layer#fact table#grain validation
One-click install
npx skills add https://github.com/databricks-solutions/vibe-coding-workshop-template --skill 04-grain-validation
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: 04-grain-validation
Source: https://github.com/databricks-solutions/vibe-coding-workshop-template/tree/main/data_product_accelerator/skills/gold/pipeline-workers/04-grain-validation
Command: npx skills add https://github.com/databricks-solutions/vibe-coding-workshop-template --skill 04-grain-validation

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill prevents costly data errors and table rewrites in the Gold layer by validating that the grain of DataFrames matches the DDL PRIMARY KEY before executing a MERGE operation.

Core Features & Use Cases

  • Pre-Merge Grain Validation: Catches mismatches between DataFrame grain and DDL PRIMARY KEY before data is merged.
  • Grain Inference: Analyzes DDL to understand the expected grain type (transaction, aggregated, snapshot).
  • Common Mistake Prevention: Provides patterns to fix issues like aggregated DDL with transaction scripts, or vice-versa.
  • Use Case: When merging a daily aggregated sales fact table, this skill ensures the incoming DataFrame is correctly grouped by date_key, store_key, and product_key as defined in the table's PRIMARY KEY, preventing duplicate or incorrectly aggregated rows.

Quick Start

Use the 04-grain-validation skill to validate the grain of the DataFrame against the 'fact_sales_daily' table's PRIMARY KEY before performing a MERGE operation.

Frequently Asked Questions about 04-grain-validation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I validate a DataFrame grain against a fact table primary key before merging?▼

To validate a DataFrame grain before merging, compare the DataFrame's grouping structure against the fact table's DDL PRIMARY KEY definitions to ensure they match and prevent merge errors.

What is fact table grain validation in a data warehouse Gold layer?▼

Fact table grain validation is the process of checking that incoming DataFrame aggregation levels match the expected transaction, aggregated, or snapshot grain defined by the target table's DDL PRIMARY KEY.

Why does my merge operation fail with duplicate rows in an aggregated sales fact table?▼

Your merge fails because the incoming DataFrame grain does not match the DDL PRIMARY KEY, causing duplicate or incorrectly aggregated rows when merging data into the Gold layer fact table.

How do I infer the grain type of a fact table from its DDL?▼

You can infer the grain type by analyzing the DDL PRIMARY KEY definitions, examining whether the composite or single-column keys indicate a transaction, aggregated, or snapshot grain structure.

Can I check for DataFrame and DDL primary key mismatches before executing a merge operation?▼

Yes, you can perform pre-merge checks to ensure the DataFrame aggregation or structure matches the target Gold layer table's grain, catching mismatches before the merge operation executes.

How do I fix an aggregated DDL mismatch with transaction scripts?▼

To fix an aggregated DDL mismatch with transaction scripts, adjust the DataFrame structure to correctly group by the composite keys defined in the table's PRIMARY KEY before merging.

Related Skills

Explore More Data & Analytics Skills→
prashsubprashsub
1

04-grain-validation

Validate DataFrame grain against fact table DDL PRIMARY KEY definitions.

Community
Intermediate
prashsubprashsub
1

02-merge-patterns

Implements PySpark MERGE patterns for Silver-to-Gold data with SCD Type 1/2.

Community
Advanced
prashsubprashsub
1

gold-layer-setup

Implement Gold layer tables from YAML schema definitions with MERGE operations.

Community
Advanced
markky21markky21

database-schema-designer

Design SQL and NoSQL database schemas with normalization and migration guidance.

Community
Basic