skills.rest
ivanshamaev
ivanshamaevCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

trino-airflow-lakehouse-pipelines

Implement Airflow-orchestrated Iceberg Lakehouse ETL workflows with Trino SQL across Bronze, Silver, and Gold layers.

14|1|Updated May 5, 2026
Search Tags:#trino#airflow#medallion architecture#iceberg#data quality gates#lakehouse etl#snapshot maintenance
One-click install
npx skills add https://github.com/ivanshamaev/de-agent-skills --skill trino-airflow-lakehouse-pipelines
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: trino-airflow-lakehouse-pipelines
Source: https://github.com/ivanshamaev/de-agent-skills/tree/main/group_skills/trino_group_skills/trino_airflow_lakehouse_pipelines
Command: npx skills add https://github.com/ivanshamaev/de-agent-skills --skill trino-airflow-lakehouse-pipelines

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps you reliably orchestrate multi-layer Lakehouse ETL (Bronze→Silver→Gold) on Iceberg tables while ensuring data quality, handling late-arriving data, and keeping table metadata healthy through scheduled maintenance.

Core Features & Use Cases

  • Medallion pipeline DAG (Bronze/Silver/Gold): Implements ingestion freshness checks, Silver merge/dedup logic, Gold aggregations, and optimizer-friendly ANALYZE steps, all executed via Airflow with Trino operators/hooks.
  • Iceberg maintenance jobs as Airflow tasks: Schedules compaction and operational housekeeping using Trino SQL for optimize, expire_snapshots, and remove_orphan_files with correct task chaining.
  • Partition backfill with dynamic task mapping: Generates partition date ranges and backfills partition-by-partition with idempotent deletes and reloads into the Silver layer.
  • Late-arriving data handling via watermark table: Tracks last processed watermark per pipeline/table and provides the pattern to query/update watermarks safely.
  • Post-load data quality gates: Enforces row-count, freshness, and null-rate checks using Trino queries (patterned with a TrinoHook-based approach), preventing promotion of bad data.

Quick Start

Ask the agent to generate an Airflow DAG for an Iceberg medallion pipeline that runs Trino-based Bronze→Silver merges, executes DQ gates per day, loads Gold aggregates, and schedules daily Iceberg maintenance jobs after completion.

Frequently Asked Questions about trino-airflow-lakehouse-pipelines

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I orchestrate Iceberg Lakehouse ETL pipelines with Airflow and Trino?▼

You can build Airflow DAGs that move data across Bronze, Silver, and Gold layers using Trino-backed SQL execution. The workflow includes ingestion freshness checks, Silver merges, Gold aggregations, and optimizer-friendly ANALYZE steps via Trino operators and hooks.

How do I schedule Iceberg table maintenance like compaction and snapshot expiration in Airflow?▼

Iceberg maintenance runs as chained Airflow tasks executing Trino SQL for optimize, expire_snapshots, and remove_orphan_files. Correct task ordering ensures compaction completes before snapshot expiration and orphan file removal.

How do I handle late-arriving data in a medallion architecture pipeline?▼

Late-arriving data is handled via a watermark tracking table that records the last processed watermark per pipeline or table. The pattern provides safe query and update operations to process only new or updated records.

Can I run partition backfills in Airflow for Iceberg Silver tables?▼

Yes, partition backfills use dynamic task mapping to generate partition date ranges and reload partitions idempotently. Each partition is processed with safe deletes and reloads into the Silver layer to prevent duplication.

How do I enforce data quality gates before promoting data to Gold layers?▼

Post-load data quality gates enforce row-count, freshness, and null-rate checks using Trino queries via a TrinoHook-based approach. Configurable DQ gate thresholds prevent bad data from being promoted to downstream Gold tables.

What is the best way to structure a medallion pipeline DAG for idempotent Iceberg workloads?▼

Structure the DAG with ingestion checks, Silver merges, Gold aggregations, and chained maintenance tasks. Idempotent partition reloads, dynamic task mapping for backfills, and configurable DQ thresholds ensure reliable, repeatable Lakehouse ETL execution.

Related Skills

Explore More Software Engineering Skills→
ivanshamaevivanshamaev
14

trino-lakehouse-platform-architect

Design Trino lakehouse architectures with Iceberg catalogs and Bronze/Silver/Gold modeling.

Community
Advanced
ivanshamaevivanshamaev
14

trino-modern-data-stack-reference-architecture

Bootstrap a Trino lakehouse with Kafka, Iceberg, dbt, Airflow, and Superset.

Community
Advanced
ivanshamaevivanshamaev
14

trino_iceberg

Optimize Apache Iceberg table design and maintenance through Trino.

Community
Advanced
ivanshamaevivanshamaev
14

trino-airflow-orchestration

Orchestrate idempotent Trino SQL pipelines in Apache Airflow with TrinoOperator.

Community
Advanced