parquet-optimization

Analyze Parquet file operations and output optimization recommendations with code templates.

2|1|Updated Oct 31, 2025

Search Tags:#streaming #statistics #compression #parquet #projection #row_group

Ready to run

npx skills add https://github.com/EmilLindfors/claude-marketplace --skill parquet-optimization

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: parquet-optimization
Source: https://github.com/EmilLindfors/claude-marketplace/tree/main/plugins/rust-data-engineering/skills/parquet-optimization
Command: npx skills add https://github.com/EmilLindfors/claude-marketplace --skill parquet-optimization

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Proactively analyzes Parquet file operations and suggests optimization improvements for compression, encoding, row group sizing, and statistics. Activates when reading or writing Parquet files or discussing Parquet performance.

Core Features & Use Cases

Compression and encoding: Recommend modern codecs like ZSTD with appropriate levels; dictionary encoding where beneficial.
Row group sizing & statistics: Set max row group size and enable statistics to improve pruning.
Column projection & streaming: Use projections and streaming for large datasets to minimize memory usage.
Batching & streaming: Prefer streaming over collect for large results.

Quick Start

Tune a Parquet writer's properties for a sample dataset and verify improved read performance with a projection.

Frequently Asked Questions about parquet-optimization

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I optimize Parquet file compression and encoding?▼

Parquet compression and encoding optimization involves selecting modern codecs like ZSTD with appropriate compression levels and applying dictionary encoding where beneficial. The Skill detects usage patterns in your read/write operations and recommends specific WriterProperties settings tailored to your dataset to reduce file size while maintaining read performance.

What's the best way to tune row group size and statistics in Parquet files?▼

Row group sizing and statistics tuning improves pruning efficiency during reads. Set max row group size based on your dataset and enable statistics collection to allow the query engine to skip irrelevant row groups. The Skill analyzes your Parquet operations and provides concrete WriterProperties templates for optimal configuration.

How do I use column projection and streaming to reduce memory usage with large Parquet datasets?▼

Column projection and streaming minimize memory footprint by reading only needed columns and processing data incrementally rather than collecting entire results. The Skill identifies when AsyncArrowWriter and ParquetRecordBatchStreamBuilder patterns are applicable and recommends projection configurations with streaming approaches for large-scale operations.

When should I use streaming over collect for Parquet file operations?▼

Streaming is preferable over collect for large datasets because it processes batches incrementally, avoiding memory exhaustion. The Skill detects your data volume and operation type, then recommends streaming-based batching strategies with actionable code templates to handle datasets that would exceed available memory if collected entirely.

Can I improve Parquet read performance without changing my write logic?▼

Yes, read performance improves through column projection and row group statistics even with unchanged writers. The Skill analyzes your existing Parquet files and read patterns, recommending projection strategies and verifying performance gains to accelerate queries without altering your write implementation.