Process out-of-core tabular datasets with Vaex's lazy DataFrames.
npx skills add https://github.com/mkurman/tamux --skill vaex-mkurman
Please help me install this Agent Skill. Skill: vaex Source: https://github.com/mkurman/tamux/tree/main/skills/scientific-skills/vaex Command: npx skills add https://github.com/mkurman/tamux --skill vaex-mkurman
Vaex enables fast, memory-efficient analysis of datasets that do not fit in RAM by providing lazy, out-of-core DataFrames that stream data from disk as needed.
Open a large dataset with vaex.open and compute a simple mean to observe lazy evaluation.
High-intent search queries and answers about installing and using this skill.
Process massive tabular datasets exceeding RAM by using lazy, out-of-core DataFrames that stream data from disk via memory mapping. This approach enables interactive analysis, fast filtering, and aggregations on billion-row datasets without loading everything into memory.
Lazy evaluation defers computations until results are explicitly requested, minimizing memory overhead. When combined with out-of-core DataFrames, it allows you to perform fast filtering and plot large-scale visualizations across terabytes of data efficiently.
Yes, out-of-core DataFrames support seamless I/O with HDF5, Arrow, and Parquet formats. This cross-format integration enables scalable analytics by allowing memory mapping and lazy evaluation directly on files stored on disk.
You can compute aggregations and virtual columns on billion-row datasets by opening the file with memory mapping and applying lazy evaluation. Virtual columns are computed on-the-fly without duplicating data, enabling efficient interactive exploration.
While out-of-core DataFrames handle terabytes of data via lazy evaluation, complex machine learning workflows may face limitations if algorithms require full in-memory dataset access or lack integration with memory-mapped virtual columns.
sagunkayasthaProcess and visualize tabular datasets larger than RAM using out-of-core DataFrames.
hxk622Process large tabular datasets exceeding RAM using out-of-core DataFrames and lazy evaluation.
ovachieverProcess and analyze out-of-core datasets exceeding RAM with Vaex DataFrames.
Zeyuyang-0420Process large tabular datasets with lazy evaluation and out-of-core operations.
