trino-airflow-lakehouse-pipelines

What problem does it solve?

This Skill helps you reliably orchestrate multi-layer Lakehouse ETL (Bronze→Silver→Gold) on Iceberg tables while ensuring data quality, handling late-arriving data, and keeping table metadata healthy through scheduled maintenance.

Core Features & Use Cases

• Medallion pipeline DAG (Bronze/Silver/Gold): Implements ingestion freshness checks, Silver merge/dedup logic, Gold aggregations, and optimizer-friendly ANALYZE steps, all executed via Airflow with Trino operators/hooks.
• Iceberg maintenance jobs as Airflow tasks: Schedules compaction and operational housekeeping using Trino SQL for optimize, expire_snapshots, and remove_orphan_files with correct task chaining.
• Partition backfill with dynamic task mapping: Generates partition date ranges and backfills partition-by-partition with idempotent deletes and reloads into the Silver layer.
• Late-arriving data handling via watermark table: Tracks last processed watermark per pipeline/table and provides the pattern to query/update watermarks safely.
• Post-load data quality gates: Enforces row-count, freshness, and null-rate checks using Trino queries (patterned with a TrinoHook-based approach), preventing promotion of bad data.

Quick Start

Ask the agent to generate an Airflow DAG for an Iceberg medallion pipeline that runs Trino-based Bronze→Silver merges, executes DQ gates per day, loads Gold aggregates, and schedules daily Iceberg maintenance jobs after completion.