skills.rest
kindofluke
kindoflukeCOMMUNITY
Explore all skills by this author →
GitHub

dataset-manager

Generate, inspect, consolidate, and optimize TPC-H parquet datasets for Sirius benchmarking.

Updated Mar 18, 2026
Search Tags:#benchmark#gpu#parquet#dataset#cudf#tpch#pyarrow
One-click install
npx skills add https://github.com/kindofluke/albatross-data --skill dataset-manager-kindofluke
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: dataset-manager
Source: https://github.com/kindofluke/albatross-data/tree/main/.vibe/skills/dataset-manager
Command: npx skills add https://github.com/kindofluke/albatross-data --skill dataset-manager-kindofluke

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Generate, inspect, consolidate, and optimize TPC-H parquet datasets for benchmarking in Sirius.

Core Features & Use Cases

  • Generate: produce TPC-H parquet files at specified scale factors for benchmarking and testing.
  • Inspect: read Parquet metadata and summarize table layouts, sizes, and row groups.
  • Consolidate/Optimize: rewrite parquet files with optimized row groups, compression, and file-splitting to meet performance targets.
  • Use Case: A performance lab needs SF100 TPCH parquet datasets with compact file sizes and predictable row-group layouts for repeatable queries.

Quick Start

Run the generate/optimize workflow to create and tune TPCH parquet data for a chosen scale factor.

Frequently Asked Questions about dataset-manager

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I generate TPC-H parquet datasets at different scale factors for benchmarking?▼

TPC-H parquet datasets are generated at specified scale factors with preserved schema and optimized, partitioned outputs. The tool automatically uses cudf when available and falls back to pyarrow on GPU OOM or import failure.

Can I use cudf for parquet optimization and automatically fall back to pyarrow on GPU OOM?▼

Parquet optimization automatically uses cudf when available and falls back to pyarrow on GPU OOM or import failure. This backend selection ensures data processing continues smoothly across different hardware environments.

What is the best way to inspect parquet metadata and row group layouts before benchmarking?▼

Inspecting parquet metadata reads file layouts, sizes, and row groups to summarize table statistics. This provides detailed per-table statistics to validate data structure before running performance benchmarks.

How do I consolidate and optimize parquet row groups for repeatable query performance?▼

Consolidate and optimize parquet row groups by rewriting files with configurable row-group sizing and compression. This produces predictable, partitioned outputs with compact file sizes that meet specific performance targets.

Does this TPC-H dataset generator preserve schema across different scale factors?▼

The TPC-H dataset generator preserves schema across different SF scales while producing optimized outputs. This ensures data consistency when generating benchmarking datasets at varying sizes from small to large scales.

Related Skills

Explore More Data & Analytics Skills→
sirius-dbsirius-db
1.0k

dataset-manager

Generate and optimize TPC-H Parquet datasets with cudf or pyarrow.

Official
Advanced
sirius-dbsirius-db
1.0k

config-optimizer

Optimize Sirius database configurations for TPC-H workloads across scale factors.

Official
Advanced
silvainfmsilvainfm
2

duckdb

Run SQL analytics directly on CSV, Parquet, and JSON files and Pandas or Polars DataFrames.

Community
Advanced
EmilLindforsEmilLindfors
2

parquet-optimization

Analyze Parquet file operations and output optimization recommendations with code templates.

Community
Advanced