Process and analyze billion-row tabular datasets with lazy out-of-core DataFrames.
npx skills add https://github.com/tamagusko/linux-cfg --skill vaex-tamagusko
Please help me install this Agent Skill. Skill: vaex Source: https://github.com/tamagusko/linux-cfg/tree/main/dotfiles/claude/skills/vaex Command: npx skills add https://github.com/tamagusko/linux-cfg --skill vaex-tamagusko
delay=True execution. - ML pipelines at scale: Apply scalers, encoders, PCA, KMeans, and scikit-learn/XGBoost/LightGBM models, then save pipeline state for production deployment. - Use Case: Convert a 50GB CSV export to HDF5 once, then explore distributions, engineer features, and train an XGBoost model on the full dataset from a laptop. ## Quick Start Use the vaex skill to open my large HDF5 file, compute summary statistics by category, and plot a heatmap of two columns.High-intent search queries and answers about installing and using this skill.
Use vaex.from_csv with chunk_size to stream the file, and pass convert='output.hdf5' to create a memory-mapped HDF5 copy. Future loads with vaex.open are instant and operations stay out-of-core.
Vaex uses lazy evaluation and memory-mapped files so data never needs to fit in RAM, while pandas loads everything into memory. Vaex also supports virtual columns that compute on the fly with zero memory overhead.
HDF5 and Apache Arrow are recommended because they are memory-mapped and load instantly. Parquet works well for distributed systems, while CSV should be converted to HDF5 for repeated use.
Yes, vaex.ml provides scalers, encoders, PCA, and KMeans, plus wrappers for scikit-learn, XGBoost, LightGBM, CatBoost, and Keras. Transformations create virtual columns, and pipeline state can be saved to JSON for deployment.
Each aggregation without delay=True makes a separate pass through the data. Pass delay=True to each aggregation and combine them with vaex.execute so all statistics compute in a single pass.
Avoid Vaex for small datasets that fit comfortably in memory, where pandas is simpler, and for workflows requiring row-by-row iteration, since Vaex is designed for vectorized column operations.
K-Dense-AIProcess and analyze billion-row tabular datasets with out-of-core DataFrames.
weisbergAggregate cross-platform social performance CSVs into benchmarks and sentiment reports.
Datus-aiProfile data quality metrics and value distributions for tabular datasets.
code-yeongyuAnalyze datasets using hybrid-engine processing with DuckDB, Polars, and Python-based statistical tools.