skills.rest
ivanshamaev
ivanshamaevCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

trino-modern-data-stack-reference-architecture

Bootstrap a Trino lakehouse with Kafka, Iceberg, dbt, Airflow, and Superset.

14|1|Updated May 5, 2026
Search Tags:#monitoring#dbt#trino#airflow#kafka#iceberg#lakehouse
One-click install
npx skills add https://github.com/ivanshamaev/de-agent-skills --skill trino-modern-data-stack-reference-architecture
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: trino-modern-data-stack-reference-architecture
Source: https://github.com/ivanshamaev/de-agent-skills/tree/main/group_skills/trino_group_skills/trino_modern_data_stack_reference_architecture
Command: npx skills add https://github.com/ivanshamaev/de-agent-skills --skill trino-modern-data-stack-reference-architecture

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill provides an end-to-end, production-oriented reference architecture for a Trino-based Modern Data Stack so you can quickly bootstrap a lakehouse platform without manually stitching together Kafka, Iceberg, Trino, dbt, Airflow, BI, and observability.

Core Features & Use Cases

  • End-to-end component wiring (single-machine Docker Compose): Kafka + MinIO/S3 + Hive Metastore + Trino + Airflow + dbt + Superset + Prometheus/Grafana, with ready-to-run configurations for how the systems connect.
  • Medallion lakehouse bootstrapping (Bronze/Silver/Gold DDL): Creates Iceberg schemas and example tables for raw ingest, cleansed typed models, and aggregated business-ready outputs.
  • Operational pipeline blueprints: Includes Airflow DAG patterns for Kafka→Bronze ingest, Bronze→Silver transformations with MERGE/dedup logic, and Gold refresh plus routine Iceberg maintenance (compaction, snapshot expiration, orphan file cleanup).
  • Transformation and serving integration: Provides a dbt project layout with Trino profiles and example models, plus Superset dataset configuration guidance and monitoring alert rules.

Quick Start

Ask the agent to deploy the full docker-compose stack, bootstrap the Iceberg Bronze/Silver/Gold schemas in Trino, run the dbt models via Airflow, and trigger the example Airflow DAGs while validating Trino/Superset/Grafana connectivity.

Frequently Asked Questions about trino-modern-data-stack-reference-architecture

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up a Trino lakehouse with Kafka and Iceberg?▼

You can set up a Trino lakehouse by deploying a single-machine Docker Compose stack that wires Kafka ingest, Iceberg medallion layers, Trino query execution, Airflow orchestration, and Superset BI into an end-to-end reference architecture.

What is the medallion architecture pattern for an Iceberg lakehouse?▼

The Iceberg medallion architecture structures data through Bronze, Silver, and Gold layers. It starts with raw ingest, moves to cleansed typed models via dbt, and finishes with aggregated business-ready outputs for BI serving.

How do I orchestrate Kafka to Iceberg ingest with Airflow?▼

You orchestrate Kafka to Iceberg ingest by using Airflow DAG patterns that trigger raw Bronze ingestion, execute Bronze to Silver transformations with MERGE and dedup logic, and refresh Gold aggregates alongside routine Iceberg maintenance.

Can I use dbt models with Trino for lakehouse transformations?▼

Yes, dbt works with Trino for lakehouse transformations. The reference architecture provides a dbt project layout with Trino profiles and example models to structure your Bronze to Silver to Gold data pipelines.

How do I perform Iceberg table maintenance like compaction and snapshot expiration?▼

You perform Iceberg table maintenance by running operational runbooks that execute compaction, snapshot expiration, and orphan file cleanup. These routine tasks are orchestrated via Airflow DAGs to maintain lakehouse performance.

Does this Trino reference architecture work on a single machine for evaluation?▼

Yes, this Trino reference architecture targets deployment on a single machine using Docker Compose. It is designed for evaluating the modern data stack, migrating from legacy Hadoop or Hive, and onboarding teams.

Related Skills

Explore More Data & Analytics Skills→
Vivekmano27Vivekmano27

env-setup

Configures a multi-service local dev stack with Docker Compose and environment variables.

Community
Advanced
CENKSSSCENKSSS

observability-testing-patterns

Automate validation of Grafana dashboards, alert rules, and APM traces.

Community
Advanced
ivanshamaevivanshamaev
14

trino-docker-compose-stack

Deploy a local Trino lakehouse with Iceberg, Hive Metastore, and MinIO.

Community
Advanced
michsindlingermichsindlinger
1

Infrastructure Provisioning Skill

Automate cloud infrastructure provisioning with Terraform across DigitalOcean, AWS, and GCP.

Community
Advanced