vaex

Out-of-core processing analyzes massive tabular datasets using lazy evaluation and memory mapping, reading data from disk instead of loading everything into RAM. This enables scalable big data exploration without memory limits.

Lazy evaluation defers computations until results are explicitly requested, allowing you to chain filtering and aggregations on large files. It processes data in chunks, ensuring minimal memory footprint during big data analysis.

Virtual columns support feature engineering on big data by defining expressions that are computed on the fly rather than materializing new arrays in memory. This works seamlessly across CSV, HDF5, and Arrow formats during exploration.

Memory-mapped I/O works directly with lazy evaluation to provide instant loading for cross-format tabular datasets. It maps files from formats like Parquet and Arrow directly into memory space, enabling scalable analysis without active RAM consumption.

Out-of-core processing is necessary when analyzing experiment logs, telemetry, or large result tables that exceed available memory limits. It allows you to process big data datasets that would otherwise cause system crashes during loading.