working-with-large-data

To process Parquet datasets larger than memory on an HPC cluster, use out-of-core processing with DuckDB or Polars to stream and chunk data. This prevents RAM exhaustion by leveraging lazy evaluation and on-disk operations.

Out-of-core processing handles large data by streaming and chunking datasets that exceed available RAM. You need it for terabyte-scale workflows where loading entire datasets into memory would cause system failure or crashes.

Yes, DuckDB works with Polars for chunked data processing on Slurm by utilizing columnar formats and Arrow integration. They enable scalable HPC workflows through lazy evaluation and on-disk operations to manage memory usage.

The best way to run batch data pipelines on Yale SOM HPC without hitting memory limits is launching Slurm jobs that read Parquet data and process it in chunked steps using DuckDB or Polars, applying lazy evaluation.

Loading large Arrow datasets exhausts RAM because eager loading pulls entire datasets into memory. Prevent this by applying out-of-core processing with Polars or DuckDB, using lazy evaluation and chunked streaming.