skills.rest
DKSang
DKSangCOMMUNITY
Explore all skills by this author →
GitHub

des-duckdb-local-engine

Profile datasets and validate schema, grain, and quality using DuckDB or DuckLake.

2|Updated May 20, 2026
Search Tags:#data quality#schema validation#duckdb#ducklake#data profiling#parquet conversion#sql prototyping
One-click install
npx skills add https://github.com/DKSang/DES-SKILL --skill des-duckdb-local-engine
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: des-duckdb-local-engine
Source: https://github.com/DKSang/DES-SKILL/tree/main/skills-stack/des-duckdb-local-engine
Command: npx skills add https://github.com/DKSang/DES-SKILL --skill des-duckdb-local-engine

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

des-duckdb-local-engine prevents downstream DES artifacts from being built on assumptions by using DuckDB (and optionally DuckLake) to inspect real data, validate schema/grain/quality, and capture evidence early in the workflow.

Core Features & Use Cases

  • Local-first source profiling: read raw files and infer/validate schema, row counts, nulls, duplicates, ranges, and categorical distributions to reduce uncertainty.
  • Grain and key validation: test candidate keys and join cardinality before drafting Silver/Gold logic so aggregation grain and relationships are trustworthy.
  • SQL prototyping and conversion: prototype transformation SQL locally, generate conversion/export plans (e.g., to Parquet), and produce artifact-ready evidence packages.
  • DuckLake local lakehouse layer: use Parquet + SQL catalog + snapshots/time travel for a lightweight local lakehouse to support Bronze/Silver/Gold prototypes and schema evolution experiments.

Quick Start

Use des-duckdb-local-engine to profile your provided data files and write findings into the matching DES evidence outputs for the active phase.

Frequently Asked Questions about des-duckdb-local-engine

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I profile local data files and validate schema before building data pipelines?▼

Profile local data files and validate schema using DuckDB to inspect real datasets, inferring schema, row counts, nulls, and categorical distributions to capture evidence early and reduce downstream uncertainty.

What is the best way to test candidate keys and join cardinality for SQL aggregation grain?▼

Testing candidate keys and join cardinality is best done locally by querying raw files with DuckDB to validate grain and relationships before drafting Silver or Gold transformation logic.

Can I use DuckLake for local lakehouse snapshots and time travel during data prototyping?▼

Yes, you can use DuckLake for a lightweight local lakehouse layer, leveraging Parquet files with a SQL catalog to support Bronze, Silver, and Gold prototypes alongside schema evolution experiments with snapshots and time travel.

How do I prototype transformation SQL locally and generate a Parquet conversion plan?▼

Prototype transformation SQL locally by running queries against raw files in DuckDB, then generate conversion and export plans to Parquet while producing artifact-ready evidence packages for pipeline design.

Does DuckDB require write access to source files for safe read-only data profiling?▼

No, DuckDB performs safe read-only sampling to inspect source files, ensuring that local data profiling and schema validation occur without modifying or risking the original raw datasets.

Related Skills

Explore More Data & Analytics Skills→
nrminornrminor

duckdb

Query Parquet, CSV, and JSON files in place with DuckDB SQL.

Community
Intermediate
erichowenserichowens
181

data-pipeline-engineer

Automate ETL/ELT pipeline design, validation, and operation across medallion architectures.

Community
Advanced
DKSangDKSang
2

des-airflow-orchestration

Orchestrate DES workflows into scheduled Apache Airflow DAGs with validation gates.

Community
Advanced
takusaotometakusaotome
5

duckdb-expert

Profile CSV, Parquet, and JSON datasets with DuckDB SQL.

Community
Advanced