dask

Use dask.dataframe.read_csv to load the file lazily in partitions instead of pandas. Operations like filtering and groupby build a task graph and only execute when you call compute(), so the full dataset never sits in memory.

DataFrames parallelize pandas for tabular data, Arrays parallelize NumPy for numeric data using blocked algorithms, and Bags handle unstructured data like JSON or logs with functional operations. Convert Bags to DataFrames once data is structured.

Use Dask when scaling existing pandas or NumPy code beyond memory or across clusters. Use vaex for out-of-core analytics on a single machine and polars for fast in-memory processing, as noted in the skill's selection guidance.

Use threads for NumPy and pandas operations that release the GIL, processes for pure Python code, synchronous for debugging with pdb, and the distributed scheduler when you need the dashboard or multi-machine clusters.

A slow start usually means the task graph is too large, often millions of tiny tasks. Increase chunk sizes toward roughly 100 MB per chunk or fuse operations with map_partitions or map_blocks to reduce task count.

Avoid Dask when data fits comfortably in memory and computations finish quickly with pandas or NumPy. First try better algorithms, efficient formats like Parquet, compiled code via Numba, or data sampling before adding distributed complexity.