duckdb

Run SQL analytics directly over Parquet, CSV, JSON, and Iceberg/Delta files.

14|1|Updated May 5, 2026

Search Tags:#data transformation #parquet #duckdb #sql querying #python api #embedded database #s3 analytics

Ready to run

npx skills add https://github.com/ivanshamaev/de-agent-skills --skill duckdb-ivanshamaev

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: duckdb
Source: https://github.com/ivanshamaev/de-agent-skills/tree/main/skills/duckdb
Command: npx skills add https://github.com/ivanshamaev/de-agent-skills --skill duckdb-ivanshamaev

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

DuckDB removes the friction of loading Parquet/CSV/JSON (and Iceberg/Delta) into a separate database by letting you run analytical SQL directly over files for fast exploration and transformation.

Core Features & Use Cases

File-first SQL analytics: Query Parquet, CSV, JSON, and partitioned datasets with predicate and projection pushdown, including Hive-style partition directories on S3.
Embedded ETL and transformation: Transform file-to-file (e.g., Parquet → Parquet) using SQL, including window functions, QUALIFY, PIVOT/UNPIVOT, and ASOF joins.
Practical ecosystem support: Use Python API for DataFrame/Arrow/Pandas workflows, and extend scans via extensions like httpfs, iceberg, delta, postgres, and mysql.
Use case: When you need to analyze a large folder of partitioned Parquet data (e.g., silver-layer events on S3) by date and region, you can filter and aggregate immediately in SQL without standing up Spark or a database.

Quick Start

Run DuckDB SQL to aggregate revenue from partitioned Parquet on S3 by executing a query that uses hive_partitioning and filters on event_date and region.

Frequently Asked Questions about duckdb

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I query Parquet files on S3 without loading them into a database?▼

DuckDB lets you run SQL directly over Parquet, CSV, and JSON files without a separate database. It applies predicate and projection pushdown to scan partitioned datasets locally or on S3, enabling fast exploration and file-to-file ETL transformations.

Can I use SQL to transform Parquet files into other formats?▼

Yes, you can embed file-to-file ETL transformations in SQL, converting Parquet, CSV, or JSON into other formats. It supports window functions, QUALIFY, PIVOT/UNPIVOT, and ASOF joins for complex transformations without a separate database.

Does DuckDB work with Python for DataFrame and Pandas workflows?▼

Yes, DuckDB provides a Python API for DataFrame, Arrow, and Pandas workflows. It uses vectorized execution to run Python-driven analytics and transformations directly over Parquet, CSV, and JSON files.

What is the best way to analyze a large folder of partitioned Parquet data?▼

The best way is to run SQL directly over the folder using Hive-style partition directory support. You can filter and aggregate immediately by date and region without standing up Spark or a separate database engine.

Do I need to install extensions to query Iceberg and Delta tables?▼

Yes, you need to load optional extensions like iceberg and delta to use iceberg_scan and delta_scan. These extensions enable direct SQL analytics over Iceberg and Delta sources without preloading into a separate database.

When should I not use an embedded SQL database for file analytics?▼

You should avoid embedded SQL file analytics when your workflows require a persistent centralized database server, or when your data processing needs exceed local in-process analytics and vectorized execution limits.