skills.rest
takusaotome
takusaotomeCOMMUNITY
Explore all skills by this author →
GitHub

duckdb-expert

Profile CSV, Parquet, and JSON datasets with DuckDB SQL.

5|4|Updated Nov 9, 2025
Search Tags:#data quality#duckdb#sql optimization#data profiling#etl pipeline#csv parquet json#large dataset analysis
One-click install
npx skills add https://github.com/takusaotome/claude-skills-library --skill duckdb-expert-takusaotome
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: duckdb-expert
Source: https://github.com/takusaotome/claude-skills-library/tree/main/skills/duckdb-expert
Command: npx skills add https://github.com/takusaotome/claude-skills-library --skill duckdb-expert-takusaotome

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires duckdb, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill helps you analyze and transform large CSV, Parquet, and JSON datasets efficiently using DuckDB, avoiding slow loading and memory issues common with traditional tooling.

Core Features & Use Cases

  • Direct File Query (SQL on files): Run complex SQL directly against CSV/Parquet/JSON (including wildcards and remote URLs via extensions).
  • Scalable Data Profiling & Quality Checks: Generate schema, statistics (optionally sampled), NULL/completeness metrics, duplicates, and sample rows for rapid understanding.
  • ETL Pipeline Template: Build extract-transform-load workflows with optional column selection, filtering, partitioning, compression, and retry-based loading.
  • Performance Optimization Guidance: Use DuckDB-specific tactics like predicate pushdown, column pruning, parallelism, and EXPLAIN/PRAGMA profiling.

Quick Start

Run the profiling script on a dataset (for example, data/*.parquet) to produce a Markdown data profile report with schema, statistics, and data quality metrics.

Frequently Asked Questions about duckdb-expert

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I query large CSV and Parquet files without loading them into memory?▼

You can query large CSV and Parquet files directly using DuckDB SQL, which processes data efficiently without slow loading or memory issues. This Skill generates SQL to run complex queries directly against local or wildcarded file paths.

What is the best way to generate data quality metrics for a huge dataset?▼

Data quality metrics for huge datasets are best generated using DuckDB SQL to compute schema, statistics, NULL completeness, and duplicate-row metrics. This Skill optionally samples large datasets to produce these profiles deterministically.

Can I use DuckDB to build an ETL pipeline from JSON files?▼

Yes, you can use DuckDB to build ETL pipelines from JSON files. This Skill provides templates for extract-transform-load workflows including column selection, filtering, partitioning, compression, and retry-based loading via read_json.

How do I optimize SQL performance when analyzing large datasets in DuckDB?▼

Optimize SQL performance in DuckDB by using tactics like predicate pushdown, column pruning, and parallelism. This Skill provides performance-aware querying guidance using EXPLAIN and PRAGMA profiling to improve large dataset analysis.

Does DuckDB work with remote data sources for exploratory data analysis?▼

Yes, DuckDB works with remote data sources for exploratory analysis by using extensions to query remote URLs. This Skill generates SQL to compute data profiles and quality metrics across wildcarded or remote sources via read_csv and read_parquet.

Related Skills

Explore More Data & Analytics Skills→
ivanshamaevivanshamaev
14

duckdb

Run SQL analytics directly over Parquet, CSV, JSON, and Iceberg/Delta files.

Community
Intermediate
richard-gyikorichard-gyiko
3

data-wrangler

Execute DuckDB SQL to transform and export data across CSV, Parquet, JSON, Excel, and databases.

Community
Intermediate
alexismanuelalexismanuel

duckdb-data-explorer

Profile CSV, Parquet, and JSON datasets with DuckDB queries.

Community
Advanced
dotlab-hqdotlab-hq

cudf-analytics

Perform GPU-accelerated groupby aggregations and statistical summaries on large CSV datasets.

Official
Intermediate