parquet-optimization

Parquet compression and encoding optimization involves selecting modern codecs like ZSTD with appropriate compression levels and applying dictionary encoding where beneficial. The Skill detects usage patterns in your read/write operations and recommends specific WriterProperties settings tailored to your dataset to reduce file size while maintaining read performance.

Row group sizing and statistics tuning improves pruning efficiency during reads. Set max row group size based on your dataset and enable statistics collection to allow the query engine to skip irrelevant row groups. The Skill analyzes your Parquet operations and provides concrete WriterProperties templates for optimal configuration.

Column projection and streaming minimize memory footprint by reading only needed columns and processing data incrementally rather than collecting entire results. The Skill identifies when AsyncArrowWriter and ParquetRecordBatchStreamBuilder patterns are applicable and recommends projection configurations with streaming approaches for large-scale operations.

Streaming is preferable over collect for large datasets because it processes batches incrementally, avoiding memory exhaustion. The Skill detects your data volume and operation type, then recommends streaming-based batching strategies with actionable code templates to handle datasets that would exceed available memory if collected entirely.

Yes, read performance improves through column projection and row group statistics even with unchanged writers. The Skill analyzes your existing Parquet files and read patterns, recommending projection strategies and verifying performance gains to accelerate queries without altering your write implementation.