Implement Delta Lake tables and tune performance with liquid clustering and Z-ordering.
npx skills add https://github.com/Wbunker/skills-repo --skill delta-lake-expert
Please help me install this Agent Skill. Skill: delta-lake-expert Source: https://github.com/Wbunker/skills-repo/tree/main/delta-lake-expert Command: npx skills add https://github.com/Wbunker/skills-repo --skill delta-lake-expert
This Skill provides expert guidance on implementing, optimizing, and managing Delta Lake, enabling robust and efficient data lakehouse operations.
Use the delta-lake-expert skill to understand how to implement liquid clustering for a new Delta table.
High-intent search queries and answers about installing and using this skill.
Delta Lake supports a Medallion architecture by organizing data into Bronze, Silver, and Gold layers. This multi-hop approach progressively enhances data quality and structure, ensuring reliable ACID transactions across each tier of your data lakehouse.
Delta Lake performance tuning optimizes query speed using techniques like liquid clustering and Z-ordering. Liquid clustering dynamically co-locates data without requiring manual partitioning, while Z-ordering rearranges data on disk to accelerate filtered queries.
Delta Lake supports streaming ingestion and Change Data Feed for building real-time pipelines. Change Data Feed tracks row-level changes between versions, enabling downstream streaming applications to process incremental updates efficiently.
Delta Lake provides ACID transactions, liquid clustering, and UniForm for cross-format compatibility. When compared to Iceberg and Hudi, it distinguishes itself through deep integration with Spark and robust MERGE/UPDATE/DELETE operations for lakehouse management.
Delta Lake implements SCD Type 2 using MERGE operations to track historical changes, while its data governance features ensure compliance. This combination maintains accurate historical records and secure, well-managed data lakehouse assets.
patrikboroschMigrate HDInsight Spark, Hive, and Oozie workloads to Microsoft Fabric.
patrikboroschAutomate Bronze/Silver/Gold medallion lakehouse pipelines in Microsoft Fabric.
personamanagmentlayerProvide Databricks expertise on Spark, Delta Lake, MLflow, and Unity Catalog.
WbunkerExplain Apache Hudi architecture, table types, operations, and tuning.
