skills.rest
AnderHonorato
AnderHonoratoCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

polars

Process and transform data with Polars DataFrames using lazy evaluation and parallel execution.

Updated Aug 28, 2026
Search Tags:#data processing#etl#parquet#polars#dataframe#lazy evaluation#pandas migration
One-click install
npx skills add https://github.com/AnderHonorato/Mem-rias-IA---Infinity --skill polars-anderhonorato
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: polars
Source: https://github.com/AnderHonorato/Mem-rias-IA---Infinity/tree/main/Manus/Skills/snapshots/kdense-polars
Command: npx skills add https://github.com/AnderHonorato/Mem-rias-IA---Infinity --skill polars-anderhonorato

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires polars, and includes references (resource) components.

What problem does it solve? Working with large datasets in Python often means slow pandas operations, high memory usage, and single-threaded bottlenecks. This Skill provides comprehensive guidance for using Polars, the Apache Arrow-based DataFrame library, to run expression-based transformations with lazy query optimization, parallel execution, and streaming for out-of-core processing. ## Core Features & Use Cases - Expression-Based Data Manipulation: Select, filter, group, aggregate, and apply window functions using composable Polars expressions that parallelize automatically. - Lazy Evaluation & Streaming: Build optimized query plans with predicate and projection pushdown, and process datasets larger than RAM via the streaming engine. - Pandas Migration: Follow detailed operation mappings and anti-pattern guidance to convert existing pandas code to faster Polars equivalents. - Use Case: You have a 20GB CSV of transaction logs. Use lazy scanning to filter and aggregate only the needed columns, then write partitioned Parquet output without loading the full file into memory. ## Quick Start Ask the AI to read a CSV file with Polars, filter rows where a column exceeds a threshold, group by a category, and compute summary statistics using lazy evaluation.

Frequently Asked Questions about polars

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I migrate from pandas to Polars?▼

Replace pandas operations with Polars equivalents: use df.filter() instead of boolean indexing, df.with_columns() instead of df.assign(), and df.group_by().agg() instead of groupby().agg(). Polars has no index, enforces strict typing, and uses .over() for window functions instead of transform().

How do I process large CSV files with Polars?▼

Use pl.scan_csv() instead of pl.read_csv() to build a lazy query plan, then filter and select columns early before calling .collect(). For datasets larger than RAM, pass engine="streaming" to collect() or use sink_parquet() for streaming writes.

What is the difference between Polars DataFrame and LazyFrame?▼

A DataFrame executes operations immediately (eager), while a LazyFrame builds a query plan that Polars optimizes before execution. LazyFrames enable predicate pushdown, projection pushdown, and streaming, making them preferable for large datasets and complex pipelines.

Does Polars support reading from S3 and cloud storage?▼

Yes, Polars reads and writes Parquet and other formats directly from S3, Azure Blob Storage, and Google Cloud Storage using URI paths like s3://bucket/file.parquet. Use credential providers such as CredentialProviderAWS or Azure DefaultAzureCredential rather than hardcoding secrets.

Why is my Polars code running slowly?▼

Common causes include using Python lambdas in map_elements (which disables parallelization), reading large files eagerly instead of lazily, and filtering after joins instead of before. Stay within the expression API, use scan_* functions, and filter and select columns early in the pipeline.

When should I not use Polars instead of pandas?▼

Stay with pandas when you rely on complex index-based time series operations, need libraries that only accept pandas DataFrames, or work with small data where performance is not critical. Polars suits large datasets, performance-critical pipelines, and new projects.

Related Skills

Explore More Data & Analytics Skills→
masermediagroup-stackmasermediagroup-stack
1

ingesting-data

Implements data ingestion pipelines from cloud storage, APIs, files, and streaming sources into databases.

Community
Advanced
Edgame2Edgame2

add-datalake-consumer

Ingest RabbitMQ event data into Azure Data Lake as partitioned Parquet files.

Community
Advanced
mikeparcewskimikeparcewski
8

data

Profile tabular datasets and validate them against JSON schemas to generate quality reports.

Community
Intermediate
TeiNamTeiNam
12

duckdb-patterns

Perform in-memory analytics on CSV, Parquet, and JSON files with DuckDB.

Community
Advanced