skills.rest
ivanshamaev
ivanshamaevCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

delta-lake

Generate Delta Lake DDL, DML, and operational commands for transactional storage.

14|1|Updated May 5, 2026
Search Tags:#merge#time travel#cdc#vacuum#spark sql#delta lake#scd2
One-click install
npx skills add https://github.com/ivanshamaev/de-agent-skills --skill delta-lake-ivanshamaev
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: delta-lake
Source: https://github.com/ivanshamaev/de-agent-skills/tree/main/skills/delta_lake
Command: npx skills add https://github.com/ivanshamaev/de-agent-skills --skill delta-lake-ivanshamaev

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Delta Lake operations are complex: you need correct table design, efficient writes and upserts, safe schema evolution, and maintenance without breaking downstream consumers. This Skill reduces those risks by providing proven Delta Lake DDL/DML and operational patterns for production data engineering.

Core Features & Use Cases

  • Table setup (DDL) & partitioning: Create managed or external Delta tables with partitioning and important table properties.
  • Data operations (DML): Append data and perform updates, deletes, and MERGE upserts, SCD Type 2, and CDC-style ingestion patterns.
  • Operational maintenance & reliability: Run OPTIMIZE (including Z-ORDER), VACUUM, use Time Travel for debugging and rollback, and recover tables with RESTORE.
  • Evolving schema safely: Apply schema evolution options and column mapping required for column rename/drop.
  • Incremental consumption: Read/write with streaming semantics and use Change Data Feed for change propagation and audit.

Quick Start

Use the delta-lake skill to create a partitioned Delta table, then MERGE a latest-staging dataset into it to upsert by primary key.

Frequently Asked Questions about delta-lake

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I perform a MERGE upsert in Delta Lake to synchronize staging data?▼

To perform a Delta Lake MERGE upsert, generate DML that matches staging records by primary key, applying updates or inserts to maintain transactional consistency on object storage.

What is the best way to handle SCD Type 2 dimension changes in Spark SQL?▼

SCD Type 2 in Spark SQL is handled by generating Delta Lake MERGE operations that track historical changes, updating active records and inserting new versions to preserve dimension history.

How do I apply CDC ingestion patterns to a Delta table with deduplicated sources?▼

Apply CDC ingestion to a Delta table by generating DML that processes deduplicated change data, using Delta Lake MERGE statements to apply inserts, updates, and deletes transactionally.

Can I rename or drop columns in Delta Lake without rewriting the entire table?▼

Yes, you can rename or drop columns in Delta Lake by enabling schema evolution and column mapping options, which allow metadata-only modifications without rewriting underlying data files.

How do I use Time Travel to debug and rollback data in a Delta table?▼

Use Delta Lake Time Travel by querying table data at a specific VERSION or TIMESTAMP, and execute RESTORE commands to rollback the table state for debugging or recovery purposes.

Do I need to run VACUUM and OPTIMIZE with Z-ORDER for Delta table maintenance?▼

Yes, run OPTIMIZE with Z-ORDER to coalesce small files and accelerate queries, followed by VACUUM with retention safeguards to remove obsolete files and manage storage costs safely.

Related Skills

Explore More Software Engineering Skills→
weiflycc-cmdweiflycc-cmd

hdinsight-migration

Migrate HDInsight Spark, Hive, and Oozie workloads to Microsoft Fabric.

Community
Advanced
yunqiqiliangyunqiqiliang
8

clickzetta-data-recovery

Recover dropped or modified ClickZetta Lakehouse tables using Time Travel and UNDROP.

Community
Advanced
wesleyosantos91wesleyosantos91
6

spark-data-engineering

Build Apache Spark batch and streaming ETL pipelines with explicit schemas and partitioned Parquet writes.

Community
Advanced
hung-phanhung-phan

lakehouse

Add ACID transactions, schema evolution, and time travel to data lakes.

Community
Advanced