skills.rest
alexismanuel
alexismanuelCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

duckdb-data-explorer

Profile CSV, Parquet, and JSON datasets with DuckDB queries.

Updated Jul 8, 2024
Search Tags:#reports#data#json#profiling#transformation#duckdb
One-click install
npx skills add https://github.com/alexismanuel/dotfiles --skill duckdb-data-explorer
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: duckdb-data-explorer
Source: https://github.com/alexismanuel/dotfiles/tree/main/.opencode/skill/duckdb-data-explorer
Command: npx skills add https://github.com/alexismanuel/dotfiles --skill duckdb-data-explorer

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires duckdb, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill enables comprehensive local data exploration using DuckDB, including automated data profiling, complex JSON handling, and interactive HTML reports for CSV, Parquet, and JSON data.

Core Features & Use Cases

  • Data Profiling and Quality Analysis: Profile data and generate quality reports with null analysis, type validation, and sample data
  • Complex JSON Handling: Deep JSON operations and transformations using DuckDB
  • Interactive HTML Reports: Generate interactive HTML reports for data exploration
  • DuckDB Query Patterns & References: Learn common patterns in duckdb SQL and JSON functions

Quick Start

  • Profile a dataset: python3 scripts/data_profiler.py data.csv --output profile.json
  • Transform JSON data: python3 scripts/json_transformer.py transform "*.json" "SELECT json_extract(data, '$.user.name') as name FROM json_data" --output transformed.parquet
  • Generate HTML report: python3 scripts/html_report_generator.py profile.json report.html
  • Open report: open report.html

Frequently Asked Questions about duckdb-data-explorer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I profile CSV and JSON data locally without cloud infrastructure?▼

Data profiling analyzes datasets for quality issues like null values, type mismatches, and cardinality. DuckDB performs this locally on CSV, Parquet, and JSON files, generating automated quality reports with null analysis, type inference, and unique counts—all without uploading data.

Can I transform and query JSON data with complex nested structures?▼

Yes. DuckDB handles deep JSON operations and transformations using SQL and JSON functions like json_extract to navigate nested structures. Multi-file JSON ingestion and querying produce flexible outputs in Parquet or JSON format.

How do I generate interactive HTML reports from data exploration?▼

Interactive HTML reports visualize data exploration results by converting profiling output and analysis results into browsable reports. DuckDB's output feeds directly into HTML report generators for data visualization and sharing.

What's the best way to handle multiple data formats in a single workflow?▼

DuckDB queries work across CSV, Parquet, and JSON simultaneously in a single analysis. This unified approach eliminates format conversion steps and enables cross-format transformations and joins within one workflow.

Do I need special tools to detect data quality issues at scale?▼

Automated profiling detects null values, validates types, and counts unique values without writing custom scripts. DuckDB's built-in functions handle this for datasets of varying sizes locally, producing structured quality reports.

Can I automate data profiling and reporting as part of a data engineering pipeline?▼

Yes. Python scripts execute DuckDB queries, apply transformations, and generate reports programmatically. Outputs in Parquet or JSON integrate into downstream data engineering workflows and quality gates.

Related Skills

Explore More Data & Analytics Skills→
winstonkoh87winstonkoh87
558

data-analysis

Ingest JSON, CSV, and Parquet files into DuckDB for SQL-based analysis.

Community
Intermediate
erichowenserichowens
181

data-pipeline-engineer

Automate ETL/ELT pipeline design, validation, and operation across medallion architectures.

Community
Advanced
spqwspqw

observable-framework-sql

Execute SQL queries on CSV, JSON, Arrow, and Parquet files with DuckDB in Observable Framework.

Community
Intermediate
milasaurusmilasaurus

ml-explore

Profile datasets for structure, distributions, and quality with narrative summaries.

Community
Intermediate