skills.rest
ivanshamaev
ivanshamaevCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

duckdb

Run SQL analytics directly over Parquet, CSV, JSON, and Iceberg/Delta files.

14|1|Updated May 5, 2026
Search Tags:#data transformation#parquet#duckdb#sql querying#python api#embedded database#s3 analytics
One-click install
npx skills add https://github.com/ivanshamaev/de-agent-skills --skill duckdb-ivanshamaev
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: duckdb
Source: https://github.com/ivanshamaev/de-agent-skills/tree/main/skills/duckdb
Command: npx skills add https://github.com/ivanshamaev/de-agent-skills --skill duckdb-ivanshamaev

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

DuckDB removes the friction of loading Parquet/CSV/JSON (and Iceberg/Delta) into a separate database by letting you run analytical SQL directly over files for fast exploration and transformation.

Core Features & Use Cases

  • File-first SQL analytics: Query Parquet, CSV, JSON, and partitioned datasets with predicate and projection pushdown, including Hive-style partition directories on S3.
  • Embedded ETL and transformation: Transform file-to-file (e.g., Parquet → Parquet) using SQL, including window functions, QUALIFY, PIVOT/UNPIVOT, and ASOF joins.
  • Practical ecosystem support: Use Python API for DataFrame/Arrow/Pandas workflows, and extend scans via extensions like httpfs, iceberg, delta, postgres, and mysql.
  • Use case: When you need to analyze a large folder of partitioned Parquet data (e.g., silver-layer events on S3) by date and region, you can filter and aggregate immediately in SQL without standing up Spark or a database.

Quick Start

Run DuckDB SQL to aggregate revenue from partitioned Parquet on S3 by executing a query that uses hive_partitioning and filters on event_date and region.

Frequently Asked Questions about duckdb

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I query Parquet files on S3 without loading them into a database?▼

DuckDB lets you run SQL directly over Parquet, CSV, and JSON files without a separate database. It applies predicate and projection pushdown to scan partitioned datasets locally or on S3, enabling fast exploration and file-to-file ETL transformations.

Can I use SQL to transform Parquet files into other formats?▼

Yes, you can embed file-to-file ETL transformations in SQL, converting Parquet, CSV, or JSON into other formats. It supports window functions, QUALIFY, PIVOT/UNPIVOT, and ASOF joins for complex transformations without a separate database.

Does DuckDB work with Python for DataFrame and Pandas workflows?▼

Yes, DuckDB provides a Python API for DataFrame, Arrow, and Pandas workflows. It uses vectorized execution to run Python-driven analytics and transformations directly over Parquet, CSV, and JSON files.

What is the best way to analyze a large folder of partitioned Parquet data?▼

The best way is to run SQL directly over the folder using Hive-style partition directory support. You can filter and aggregate immediately by date and region without standing up Spark or a separate database engine.

Do I need to install extensions to query Iceberg and Delta tables?▼

Yes, you need to load optional extensions like iceberg and delta to use iceberg_scan and delta_scan. These extensions enable direct SQL analytics over Iceberg and Delta sources without preloading into a separate database.

When should I not use an embedded SQL database for file analytics?▼

You should avoid embedded SQL file analytics when your workflows require a persistent centralized database server, or when your data processing needs exceed local in-process analytics and vectorized execution limits.

Related Skills

Explore More Data & Analytics Skills→
MapleTechLabsMapleTechLabs
1.6k

chdb-sql

Execute ClickHouse SQL queries on local files and remote databases in Python.

Official
Intermediate
silversteinsilverstein
21

vaex

Process and analyze out-of-core tabular datasets with lazy DataFrames.

Community
Intermediate
nq-rdlnq-rdl

s3-explore

Query S3-compatible object storage metadata and preview Parquet files without downloading.

Official
Intermediate
tankygranny05tankygranny05
2

update-clickhouse-m3u

Update ClickHouse transactions table with incremental trading data via SSH.

Community
Intermediate