skills.rest
developer-advocacy-dremio
developer-advocacy-dremioOFFICIAL
Explore all skills by this author →
WebsiteGitHub

Dremio Iceberg Operations

Manage Iceberg tables with ACID DML operations in Dremio.

Updated Mar 6, 2026
Search Tags:#merge#partitioning#iceberg#schema-evolution#time-travel#dremio#table-management
One-click install
npx skills add https://github.com/developer-advocacy-dremio/dremio-skills-bundle --skill dremio-iceberg-operations
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: Dremio Iceberg Operations
Source: https://github.com/developer-advocacy-dremio/dremio-skills-bundle/tree/main/dremio-iceberg-skill
Command: npx skills add https://github.com/developer-advocacy-dremio/dremio-skills-bundle --skill dremio-iceberg-operations

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill removes uncertainty and manual complexity when creating, modifying, querying, and maintaining Apache Iceberg tables in Dremio, enabling reliable ACID DML, snapshot time travel, and schema/partition evolution without accidental data loss.

Core Features & Use Cases

  • DML and Upserts: Execute INSERT, UPDATE, DELETE, and MERGE operations for transactional upserts and change data workflows.
  • Schema & Partition Evolution: Add, drop, or rename columns and change partitioning strategies without rewriting historic data.
  • Time Travel & Versioned Catalogs: Query past snapshots, branch and tag catalogs (Nessie/Arctic), and rollback to prior table states for safe experimentation.
  • Maintenance & Performance: Run OPTIMIZE to compact small files and VACUUM to remove expired snapshots and orphan files.
  • Use Case: Implement an incremental ingestion pipeline that COPY INTO a staging table, MERGEs into the production Iceberg table, then OPTIMIZE and VACUUM on a schedule to maintain performance and storage hygiene.

Quick Start

Ask the skill to create an Iceberg table, load data from object storage, perform a MERGE upsert from a staging table, and then run OPTIMIZE and VACUUM to compact files and reclaim storage.

Frequently Asked Questions about Dremio Iceberg Operations

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I perform a MERGE upsert on an Apache Iceberg table in Dremio?▼

To perform a MERGE upsert on an Iceberg table in Dremio, use ACID DML operations to synchronize data from a staging table into the production table, enabling reliable change data workflows without accidental data loss.

Can I query past snapshots of Iceberg tables using time travel in Dremio?▼

Yes, you can query past snapshots of Iceberg tables using time travel in Dremio. This allows you to query historical data states, create branches and tags in versioned catalogs like Nessie, and rollback to prior table states for safe experimentation.

How do I handle schema evolution and partitioning changes in Dremio Iceberg tables?▼

You handle schema evolution in Dremio Iceberg tables by adding, dropping, or renaming columns and changing partitioning strategies without rewriting historic data, ensuring seamless structural updates as your data model evolves.

What is the best way to maintain Iceberg table performance and storage hygiene in Dremio?▼

The best way to maintain Iceberg table performance in Dremio is running OPTIMIZE to compact small files and VACUUM to remove expired snapshots and orphan files, ensuring optimal query performance and storage hygiene.

Does Dremio support building incremental ingestion pipelines for Iceberg tables?▼

Yes, Dremio supports building incremental ingestion pipelines for Iceberg tables by using COPY INTO a staging table, executing MERGE operations into production, and scheduling OPTIMIZE and VACUUM maintenance routines.

Which lakehouse catalogs can I use for Iceberg table management in Dremio?▼

You can use Dremio lakehouse catalogs like Nessie, Arctic, AWS Glue, or Open Catalog for managing, evolving, and time-traveling Iceberg tables, including branch and tag catalog operations for production workloads.

Related Skills

Explore More Data & Analytics Skills→
ivanshamaevivanshamaev
14

trino-lakehouse-platform-architect

Design Trino lakehouse architectures with Iceberg catalogs and Bronze/Silver/Gold modeling.

Community
Advanced
flag3flag3
1

postgres-patterns

Provide PostgreSQL query, schema, indexing, and security patterns aligned with Supabase best practices.

Community
Intermediate
vivekganavivekgana
4

data-classification

Automate schema- and content-based data classification and tagging across Unity Catalog.

Community
Advanced
WbunkerWbunker

apache-iceberg-expert

Explain Apache Iceberg architecture, metadata layers, and catalog management.

Community
Advanced