dataset-manager

Generate, inspect, consolidate, and optimize TPC-H parquet datasets for Sirius benchmarking.

Updated Mar 18, 2026

Search Tags:#benchmark #gpu #parquet #dataset #cudf #tpch #pyarrow

Ready to run

npx skills add https://github.com/kindofluke/albatross-data --skill dataset-manager-kindofluke

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: dataset-manager
Source: https://github.com/kindofluke/albatross-data/tree/main/.vibe/skills/dataset-manager
Command: npx skills add https://github.com/kindofluke/albatross-data --skill dataset-manager-kindofluke

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Generate, inspect, consolidate, and optimize TPC-H parquet datasets for benchmarking in Sirius.

Core Features & Use Cases

Generate: produce TPC-H parquet files at specified scale factors for benchmarking and testing.
Inspect: read Parquet metadata and summarize table layouts, sizes, and row groups.
Consolidate/Optimize: rewrite parquet files with optimized row groups, compression, and file-splitting to meet performance targets.
Use Case: A performance lab needs SF100 TPCH parquet datasets with compact file sizes and predictable row-group layouts for repeatable queries.

Quick Start

Run the generate/optimize workflow to create and tune TPCH parquet data for a chosen scale factor.

Frequently Asked Questions about dataset-manager

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I generate TPC-H parquet datasets at different scale factors for benchmarking?▼

TPC-H parquet datasets are generated at specified scale factors with preserved schema and optimized, partitioned outputs. The tool automatically uses cudf when available and falls back to pyarrow on GPU OOM or import failure.

Can I use cudf for parquet optimization and automatically fall back to pyarrow on GPU OOM?▼

Parquet optimization automatically uses cudf when available and falls back to pyarrow on GPU OOM or import failure. This backend selection ensures data processing continues smoothly across different hardware environments.

What is the best way to inspect parquet metadata and row group layouts before benchmarking?▼

Inspecting parquet metadata reads file layouts, sizes, and row groups to summarize table statistics. This provides detailed per-table statistics to validate data structure before running performance benchmarks.

How do I consolidate and optimize parquet row groups for repeatable query performance?▼

Consolidate and optimize parquet row groups by rewriting files with configurable row-group sizing and compression. This produces predictable, partitioned outputs with compact file sizes that meet specific performance targets.

Does this TPC-H dataset generator preserve schema across different scale factors?▼

The TPC-H dataset generator preserves schema across different SF scales while producing optimized outputs. This ensures data consistency when generating benchmarking datasets at varying sizes from small to large scales.