dask

To scale pandas workflows beyond RAM, use Dask DataFrames to partition large tabular datasets into chunks and execute operations lazily through parallel task graphs, calling compute at the end to return results.

Out-of-core analytics processes datasets larger than memory by operating on data chunks rather than loading everything at once. You need it for multi-file ETL, large aggregations, and chunked numerical computation exceeding available RAM.

Yes, you can scale NumPy array computations across multiple cores and machines using Dask Arrays, which partition large arrays into chunks and execute numerical operations in parallel through distributed task graphs.

Choose a scheduler based on your workload: threads for numerical work, processes for pure Python code, synchronous for debugging, or a distributed client for multi-machine execution and larger-than-RAM analytics.

Yes, lazy evaluation works well for multi-file ETL pipelines by constructing a task graph as you define transformations without executing them, then running compute to execute the entire pipeline in parallel without loading all files into memory.

Repeated compute calls slow down large-scale data processing because they re-execute the entire task graph from scratch each time instead of reusing intermediate results, making minimal compute calls and proper chunk sizing critical performance best practices.