skills.rest
gestaosian
gestaosianOFFICIAL
Explore all skills by this author →
GitHub

sian-iceberg-setup

Configure BigQuery native or Iceberg Managed ingestion formats for SIAN Raw layer tables.

Updated May 21, 2026
Search Tags:#gcp#data ingestion#BigQuery#iceberg#dag#table format#hard delete
One-click install
npx skills add https://github.com/gestaosian/gtsi-claude-skills --skill sian-iceberg-setup-gestaosian
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: sian-iceberg-setup
Source: https://github.com/gestaosian/gtsi-claude-skills/tree/main/plugins/gtsi-ops-plugin/skills/sian-iceberg-setup
Command: npx skills add https://github.com/gestaosian/gtsi-claude-skills --skill sian-iceberg-setup-gestaosian

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Choosing the wrong table ingestion format in the SIAN data platform leads to unnecessary costs or scalability problems. This Skill guides the decision between native BigQuery (WRITE_TRUNCATE full reloads) and Apache Iceberg Managed (incremental MERGE), and provides the correct connection, storage URI, clustering, and hard delete configuration. ## Core Features & Use Cases - Format Decision Matrix: Compares native vs. Iceberg formats by volume, load pattern, hard delete needs, and history preservation. - Factory Parameter Guidance: Documents required parameters like table_format="iceberg" and cluster_by, with auto-calculated connection_id and storage_uri values. - Hard Delete Handling: Explains when and how to configure hard_delete_handler.py for sources that physically remove records without a deletion flag. - Use Case: When adding a large Protheus table like SC1 with continuous growth and physical deletes at the source, use this Skill to configure an Iceberg DAG with proper clustering on D_E_L_E_T_ and the primary key. ## Quick Start Ask the AI to configure Iceberg ingestion for a new SIAN Raw layer table, including cluster columns and hard delete handling.

Frequently Asked Questions about sian-iceberg-setup

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I choose between BigQuery native and Iceberg table formats?▼

Choose native format for small or medium tables under roughly 10 million rows with daily full reloads via WRITE_TRUNCATE. Choose Iceberg for large or continuously growing tables that need incremental MERGE loads by watermark and hard delete support.

How do I configure an Iceberg table in a SIAN DAG factory?▼

Set table_format="iceberg" in the factory call and provide cluster_by with up to 4 columns. The factory automatically calculates connection_id as gcp-sian-dados.us-east1.iceberg-conn and storage_uri under gs://gcp-sian-dados-iceberg/raw/.

When do I need a hard delete handler for Iceberg tables?▼

You need hard_delete_handler.py when the source system physically removes records without marking them with a deletion flag, so they simply disappear from the API. Configure it with the project, dataset, table, source keys, and key column.

Why does my Iceberg table not appear in the BigQuery console?▼

Iceberg tables appear as External table (BigLake) entries in the BigQuery console, which is normal behavior. They are backed by the BigLake connection and Cloud Storage bucket rather than native BigQuery storage.

Why do I get duplicated data after rerunning an Iceberg DAG?▼

Duplicates after a rerun usually mean the watermark was not saved after the MERGE operation completed. Check the watermark persistence logic in the DAG factory configuration to ensure incremental state is recorded.

What are the limitations of cluster_by in Iceberg tables?▼

cluster_by accepts a maximum of 4 columns and should include columns used in frequent filters, such as deletion flags, primary keys, or reference dates. Proper clustering reduces scan costs in downstream Silver layer dbt queries.

Related Skills

Explore More Data & Analytics Skills→
DKSangDKSang
2

des-persona-source-domain-analyst

Classify source system behavior and document time semantics for data ingestion.

Community
Intermediate
microsoftmicrosoft
609

reactor-dsl

Verify Reactor API hooks, modifiers, factories, and enums against the canonical signatures index.

Official
Intermediate
memadorememadore

migrating-dbt-project-across-platforms

Migrate dbt projects across data platforms using dbt Fusion compilation.

Community
Advanced
ivanshamaevivanshamaev
14

trino-dbt-query-performance

Optimize dbt-generated SQL performance on Trino with materializations and incremental strategies.

Community
Advanced