Process in-memory DataFrames with lazy evaluation and parallel execution.
npx skills add https://github.com/Clay-HHK/claude-config --skill polars-clay-hhk
Please help me install this Agent Skill. Skill: polars Source: https://github.com/Clay-HHK/claude-config/tree/main/skills/polars Command: npx skills add https://github.com/Clay-HHK/claude-config --skill polars-clay-hhk
Polars eliminates slow, single-threaded DataFrame operations by providing a high-performance, parallel, Apache Arrow-backed in-memory engine so users can process and transform large datasets that fit in RAM with predictable typing and faster execution than pandas.
Scan the input file lazily, filter rows where age is greater than 25, select the name and age columns, and collect the result into memory.
High-intent search queries and answers about installing and using this skill.
To speed up pandas DataFrame processing for large datasets in RAM, migrate your pipelines to an Apache Arrow-backed engine that uses lazy evaluation and parallel execution. This approach minimizes runtime for ETL and analytics tasks on datasets up to tens of gigabytes.
The best way to process Parquet files for ETL is to use lazy scanning with predicate and projection pushdown. This allows the engine to filter rows and select columns during the scan, collecting only the required data into memory efficiently.
Yes, you can perform group-by aggregations and window functions on a 50GB dataset using in-memory processing. The engine handles datasets in the 1–100GB range through parallel execution and streaming collection to manage memory efficiently.
Lazy evaluation works with CSV and JSON I/O for feature engineering pipelines. The engine supports efficient reading and writing for these formats, applying expression-based transformations and type casting before collecting the final results into memory.
An Apache Arrow-backed DataFrame provides predictable typing and faster execution than standard pandas for nightly data transformations. It eliminates single-threaded operations by utilizing a high-performance parallel engine for large datasets that fit in RAM.
ejolietPerform fast, memory-efficient DataFrame operations on large datasets with Polars.
ovachieverProcess and analyze out-of-core datasets exceeding RAM with Vaex DataFrames.
brahiamsinhoOrchestrate multi-agent ML pipelines across data engineering, model training, deployment, and monitoring.
Yezez9Provide a fast in-memory DataFrame library with parallel execution and Apache Arrow backend.
