skills.rest
m19c
m19cCOMMUNITY
Explore all skills by this author →
𝕏@m19c_GitHub

databricks-iceberg

Manage Apache Iceberg tables on Databricks including Delta conversion and Snowflake interoperability.

Updated Mar 23, 2024
Search Tags:#Snowflake#Databricks#Apache Iceberg#Iceberg REST Catalog#UniForm#external engine
One-click install
npx skills add https://github.com/m19c/dotfiles --skill databricks-iceberg-m19c
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-iceberg
Source: https://github.com/m19c/dotfiles/tree/main/claude/.claude/skills/databricks-iceberg
Command: npx skills add https://github.com/m19c/dotfiles --skill databricks-iceberg-m19c

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires databricks, unity-catalog, snowflake, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill provides a comprehensive framework for working with Apache Iceberg tables on Databricks, offering managed tables, interoperability with Delta, and seamless integration with external systems.

Core Features & Use Cases

  • Managed Iceberg Tables: Create and manage native Iceberg tables with full read/write capabilities.
  • Delta to Iceberg: Convert Delta tables to Iceberg format for external engine compatibility.
  • Compatibility Mode: Enable UniForm for streaming tables and materialized views in SDP pipelines.
  • Iceberg REST Catalog (IRC): Expose Databricks tables to external engines via the REST API.
  • Snowflake Integration: Bi-directional data sharing with Snowflake Iceberg tables.
  • External Engine Access: Connect PyIceberg, OSS Spark, and other external tools for Iceberg operations.
  • Quick Start: Learn how to create a managed Iceberg table and enable UniForm on a Delta table.

Quick Start

Use the databricks-iceberg skill to create a managed Iceberg table named 'my_table' with partitioning on 'event_date'.

Frequently Asked Questions about databricks-iceberg

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I convert Delta tables to Apache Iceberg format on Databricks?▼

To convert Delta tables to Apache Iceberg format on Databricks, you enable UniForm for streaming tables and materialized views in SDP pipelines. This provides external engine compatibility, allowing your existing Delta tables to be read by other Iceberg-compatible tools.

Can I access Databricks Iceberg tables from Snowflake?▼

Yes, you can access Databricks Iceberg tables from Snowflake using bi-directional data sharing with Snowflake Iceberg tables. This integration allows seamless interoperability between Databricks and Snowflake for unified data access and analytics.

Do I need Unity Catalog to create managed Iceberg tables in Databricks?▼

Yes, you need Unity Catalog and appropriate Databricks configurations for external access to create managed Iceberg tables. Unity Catalog provides the centralized metastore management required to support native Iceberg table operations.

What is the Iceberg REST Catalog used for in Databricks?▼

The Iceberg REST Catalog (IRC) is used to expose Databricks tables to external engines via a REST API. This mechanism allows external tools like PyIceberg and OSS Spark to connect and perform Iceberg operations on your managed tables.

How do I connect external engines like PyIceberg and OSS Spark to Databricks Iceberg tables?▼

You connect external engines like PyIceberg and OSS Spark to Databricks Iceberg tables by utilizing the Iceberg REST Catalog (IRC). The IRC exposes your Databricks tables via a REST API, enabling external tools to perform standard Iceberg operations.

When should I use UniForm for Delta to Iceberg conversion?▼

You should use UniForm for Delta to Iceberg conversion when you need external engine compatibility for your streaming tables and materialized views in SDP pipelines. It enables Delta tables to be read as Iceberg without altering the primary data format.

Related Skills

Explore More Data & Analytics Skills→
mkgs-databricks-demosmkgs-databricks-demos

databricks-iceberg

Manage and access Apache Iceberg tables within Databricks.

Official
Advanced
aaronachermannaaronachermann

databricks-iceberg

Create Databricks Iceberg tables with UniForm and IRC for external engine reads.

Community
Advanced
JingyiWngJingyiWng

databricks-iceberg

Create and manage Apache Iceberg tables on Databricks with REST Catalog.

Community
Advanced
WbunkerWbunker

delta-lake-expert

Implement Delta Lake tables and tune performance with liquid clustering and Z-ordering.

Community
Advanced