polars

Replace df["col"] with df.select("col"), boolean indexing with df.filter(pl.col("x") > 10), df.assign with df.with_columns, and groupby().transform() with aggregations using .over("group"). Polars has no index, so remove set_index and reset_index calls.

Use lazy evaluation (scan_csv, LazyFrame) for large datasets, complex pipelines, or when only some columns and rows are needed, since it enables predicate and projection pushdown. Use eager DataFrames for small data and interactive exploration.

Polars is generally faster due to its Apache Arrow columnar backend, automatic parallelization, and query optimization in lazy mode. It suits 1-100GB datasets that fit in RAM; for larger-than-RAM data, consider dask or vaex instead.

Yes. Polars reads and writes Parquet and other formats directly from S3, Azure Blob, and Google Cloud Storage using URI paths with environment credentials. It also queries PostgreSQL, MySQL, and SQLite via read_database_uri.

Python functions passed to map_elements disable parallelization and run sequentially. Rewrite the logic using native Polars expressions like arithmetic on pl.col() or when/then/otherwise, which execute in parallel across threads.

Use lazy mode with streaming: call pl.scan_csv or pl.scan_parquet, build your query, then run collect(streaming=True) to process data in chunks. Alternatively, use sink_parquet to stream results directly to disk.