skills.rest
linus-mcmanamey
linus-mcmanameyCOMMUNITY
Explore all skills by this author →
GitHub

schema-reference

Validate PySpark ETL schemas against DuckDB metadata and data dictionaries.

Updated Aug 27, 2026
One-click install
npx skills add https://github.com/linus-mcmanamey/multi-agent-user-story-development --skill schema-reference
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: schema-reference
Source: https://github.com/linus-mcmanamey/multi-agent-user-story-development/tree/main/.claude/skills/schema-reference
Command: npx skills add https://github.com/linus-mcmanamey/multi-agent-user-story-development --skill schema-reference

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires duckdb, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill prevents common schema-related errors and ensures business logic compliance when generating PySpark ETL code. It automates the process of querying actual schemas, extracting business rules from data dictionaries, and comparing schemas between data layers, guaranteeing that generated code is accurate and robust.

Core Features & Use Cases

  • Dynamic Schema Querying: Retrieve exact column names, data types, and constraints from DuckDB warehouse.
  • Business Logic Extraction: Parse data dictionary files to understand relationships, default values, and data quality rules.
  • Cross-Layer Schema Comparison: Identify differences and required transformations between Bronze, Silver, and Gold layer schemas.
  • Use Case: Before writing a new PySpark transformation for a Silver layer table, use this skill to query the Bronze source schema, extract relevant business rules from the data dictionary, and compare it against the target Silver schema to ensure all transformations are correctly defined.

Quick Start

Explain the steps to create a new Silver layer table named 's_customer_case' from 'bronze_cms.b_customer_case', ensuring all schema and business logic are correctly applied.

Frequently Asked Questions about schema-reference

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I validate PySpark column names and data types against my actual schema?▼

Schema validation retrieves exact column metadata from your DuckDB warehouse to verify that PySpark transformation code uses correct column names and data types. This prevents runtime errors and ensures generated code matches your source and target schemas before execution.

Can I automatically extract business rules from a data dictionary and apply them to PySpark ETL code?▼

Yes. This Skill parses data dictionary files to extract relationships, default values, and data quality rules, then integrates those business logic constraints directly into generated PySpark transformation code.

How do I compare schemas across Bronze, Silver, and Gold layers to identify required transformations?▼

Cross-layer schema comparison queries DuckDB to identify differences between Bronze, Silver, and Gold layer schemas, then maps required transformations and column mappings so your PySpark code handles all conversions accurately.

What do I need to set up before generating PySpark ETL code with schema validation?▼

You need a DuckDB warehouse with accessible schema metadata, data dictionary files documenting business rules and column relationships, and source schemas defined for each data layer you plan to transform.

Does schema validation work with legacy data sources in PySpark transformations?▼

Yes. This Skill applies schema querying and validation proactively across Bronze, Silver, Gold, and legacy data sources, ensuring accurate column mapping and naming convention standardization regardless of source age or structure.

Why should I use automated schema validation instead of manually writing PySpark transformations?▼

Automated schema validation eliminates manual column discovery, prevents data type mismatches, ensures business rule compliance, and aligns primary and foreign keys across layers—reducing errors and development time in complex ETL pipelines.

Related Skills

Explore More Data & Analytics Skills→
AradotsoAradotso
7

ktx-data-agent-context-layer

Build a semantic context layer for AI agents querying data warehouses.

Official
Advanced
AradotsoAradotso
5

retail-etl-pipeline-medallion

Automate retail data warehouse setup with Medallion Architecture layers.

Official
Advanced
AngelP17AngelP17
1

ml-pipeline-workflow

Automate end-to-end ML pipelines from data ingestion to deployment.

Community
Advanced
shekerkammashekerkamma

archaeology

Retrieve production-tested SQL query patterns and reusable Common Table Expressions.

Community
Basic