skills.rest
manusabbath-arch
manusabbath-archCOMMUNITY
Explore all skills by this author →
GitHub

delta-lakehouse

Builds ETL pipelines that store structured and unstructured data in versioned Delta lakehouse tables.

Updated Sep 11, 2026
Search Tags:#data pipeline#etl#duckdb#lakehouse#time travel#deltalake#pyarrow
One-click install
npx skills add https://github.com/manusabbath-arch/hermes-skills --skill delta-lakehouse-manusabbath-arch
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: delta-lakehouse
Source: https://github.com/manusabbath-arch/hermes-skills/tree/main/skills/delta-lakehouse
Command: npx skills add https://github.com/manusabbath-arch/hermes-skills --skill delta-lakehouse-manusabbath-arch

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires deltalake, duckdb, pyarrow.

What problem does it solve? Operational data scattered across SQLite files, JSON logs, and APIs is unversioned and impossible to reproduce, making it hard to audit what state an AI agent saw or which dataset trained a model. This Skill turns that data into a versioned Delta lakehouse with operation history and time travel. ## Core Features & Use Cases - Heterogeneous Extraction: Normalize JSONL, CSV, API/WebSocket, and database sources into Arrow tables, keeping unstructured payloads as list/struct columns instead of forcing flat schemas. - Versioned Delta Tables: Write with write_deltalake, inspect history() for every operation, and use time travel to reproduce the exact dataset a model trained on. - Optimization & Consumption: Partition by cut columns, run VACUUM and compaction with explicit retention, and let DuckDB, ML models, and agents read the same table directly via delta_scan. - Use Case: A trading bot's prediction events live in unversioned JSON logs; ingest them into a Delta table partitioned by day, then let an analytics agent query the identical table with DuckDB while time travel reproduces any historical state. ## Quick Start Ask the agent to ingest a JSONL events file into a Delta table using deltalake, duckdb, and pyarrow, then verify the table version and query it with delta_scan.

Frequently Asked Questions about delta-lakehouse

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I write data to a Delta table in Python?▼

Use write_deltalake from the deltalake package with a pyarrow Table built via pa.Table.from_pylist. Each write creates a new table version, and DeltaTable(path).history() shows the operation log for every commit.

How do I query Delta tables with DuckDB?▼

DuckDB reads Delta tables natively using the delta_scan function, for example SELECT * FROM delta_scan('path/to/table'). It runs in memory without a server, so agents and analysts query the same table the pipeline writes.

Does deltalake support time travel to older versions?▼

Yes, DeltaTable(path, version=N) opens the table at a specific historical version. This reproduces the exact dataset that trained a model or the state an agent observed at that point.

Can I store unstructured JSON data in Delta tables?▼

Yes, carry unstructured payloads like API events as Arrow list or struct columns, or as VARIANT in DuckDB, instead of forcing them into flat columns. Normalize to a pyarrow Table during ingestion and write it directly.

Does deltalake work with S3 or cloud object storage?▼

Yes, write_deltalake and DeltaTable accept object storage URIs for S3, GCS, and Azure, with credentials passed through storage_options. The lakehouse is not limited to local disk.

When should I run VACUUM on a Delta table?▼

Run DeltaTable.vacuum() on tables with many versions to delete old partfiles, but set explicit retention first. Never vacuum data that an agent still needs to reproduce, since time travel depends on retained versions.

Related Skills

Explore More Data & Analytics Skills→
aminojaghaminojagh

create-filesystem-pipeline

Creates a dlt pipeline that loads CSV, Parquet, or JSONL files from local or cloud storage into a destination.

Community
Advanced
ivanshamaevivanshamaev
14

duckdb

Run SQL analytics directly over Parquet, CSV, JSON, and Iceberg/Delta files.

Community
Intermediate
adamregeradamreger
2

database-migrations

Implements safe, reversible database schema migrations across PostgreSQL, MySQL, and common ORMs.

Community
Intermediate
personamanagmentlayerpersonamanagmentlayer
41

databricks-expert

Provide Databricks expertise on Spark, Delta Lake, MLflow, and Unity Catalog.

Official
Advanced