skills.rest
prashsub
prashsubCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

04-grain-validation

Validate DataFrame grain against fact table DDL PRIMARY KEY definitions.

1|Updated Jan 22, 2026
Search Tags:#data modeling#data validation#etl#databricks#grain#fact table
One-click install
npx skills add https://github.com/prashsub/vibe_coding_lakehouse_starter_repo --skill 04-grain-validation-prashsub
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: 04-grain-validation
Source: https://github.com/prashsub/vibe_coding_lakehouse_starter_repo/tree/main/data_product_accelerator/skills/gold/pipeline-workers/04-grain-validation
Command: npx skills add https://github.com/prashsub/vibe_coding_lakehouse_starter_repo --skill 04-grain-validation-prashsub

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill prevents costly data errors and table rewrites by ensuring that the grain (granularity) of DataFrames being merged into fact tables precisely matches the table's defined PRIMARY KEY.

Core Features & Use Cases

  • Pre-Merge Validation: Catches grain mismatches before the MERGE operation, avoiding runtime errors like DELTA_MERGE_UNRESOLVED_EXPRESSION.
  • Grain Inference: Automatically infers the expected grain type (transaction, aggregated, snapshot) from the table's DDL.
  • Use Case: When creating a new Gold layer fact table merge script, use this Skill to automatically validate that your aggregated DataFrame's grain (e.g., daily sales) matches the DDL's composite primary key (e.g., date_key, store_key, product_key) before the merge.

Quick Start

Use the 04-grain-validation skill to validate the grain of the DataFrame before merging it into the fact_sales_daily table using the primary key columns date_key, store_key, and product_key.

Frequently Asked Questions about 04-grain-validation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I validate DataFrame grain against a Databricks fact table primary key before merging?▼

To validate DataFrame grain, this Skill compares your DataFrame schema against the Databricks fact table's DDL PRIMARY KEY definition, ensuring the grain accurately corresponds before executing the merge operation.

Why does my Databricks fact table merge fail with grain mismatch errors?▼

Your fact table merge fails because the grain of your transaction-level DataFrame does not match the aggregated data defined by the table's DDL PRIMARY KEY, causing unresolved expression errors during the merge operation.

Can I infer the expected grain type directly from a fact table DDL in Databricks?▼

Yes, you can infer the expected grain type, such as transaction, aggregated, or snapshot, directly by analyzing the fact table's DDL PRIMARY KEY definition to ensure data integrity in the Gold layer.

What is the best way to prevent runtime errors when merging aggregated data into a Gold layer fact table?▼

The best way to prevent runtime errors is to perform pre-merge validation that catches grain mismatches between the DataFrame and the DDL, avoiding table rewrites and ensuring data integrity for Gold layer fact tables.

Does this grain validation approach work for composite primary keys in Databricks ETL pipelines?▼

Yes, this grain validation works for composite primary keys by utilizing DDL analysis and DataFrame schema comparison to verify that aggregated data grain matches composite keys like date_key, store_key, and product_key.

Related Skills

Explore More Data & Analytics Skills→
databricks-solutionsdatabricks-solutions
5

04-grain-validation

Validate DataFrame grain against Gold layer DDL PRIMARY KEY definitions before merge operations.

Official
Intermediate
prashsubprashsub
1

gold-layer-setup

Implement Gold layer tables from YAML schema definitions with MERGE operations.

Community
Advanced
prashsubprashsub
1

05-schema-validation

Validate DataFrame columns against target Delta table DDL schemas before merge operations.

Community
Intermediate
diShine-digital-agencydiShine-digital-agency
1

shine-data-query

Run SQL queries on local CSV, Parquet, JSON, and Excel files.

Community
Intermediate