Scale ML data preprocessing and batch inference with Ray Data across distributed workers.
npx skills add https://github.com/qcmuu/AI-Research-Skills --skill ray-data-qcmuu
Please help me install this Agent Skill. Skill: ray-data Source: https://github.com/qcmuu/AI-Research-Skills/tree/main/05-data-processing/ray-data Command: npx skills add https://github.com/qcmuu/AI-Research-Skills --skill ray-data-qcmuu
Ray Data solves slow, memory-bound data preprocessing and batch inference by distributing streaming dataset operations across CPU/GPU clusters.
Use this for a real scenario like multi-modal batch inference where you load images from object storage, run GPU-accelerated preprocessing, and write processed outputs back to Parquet for later training.
Use the ray-data skill to read Parquet data from an S3 path, apply a batch transformation that lowercases text fields, and stream the resulting batches for inspection.
High-intent search queries and answers about installing and using this skill.
Distributed data preprocessing handles datasets larger than RAM by using Ray Data's lazy, block-based streaming execution across distributed CPU or GPU clusters.
You can apply batch transformations to Parquet data by reading the files from storage like S3 and using the map_batches operator for fast, vectorized ETL operations on streaming blocks.
Ray Data supports multi-modal loading like reading images from object storage and executing optional GPU-accelerated preprocessing for batch inference workloads.
Ray Data integrates with Ray Train to provide scalable training-time data ingestion for PyTorch and TensorFlow workflows, enabling sharded dataset handling across workers.
Running distributed ETL pipelines on large datasets is best handled by Ray Data, which applies fast vectorized batch transforms, filtering, grouping, and custom aggregations across workers.
Ray Data pipelines support reading and writing multi-format I/O including Parquet, CSV, JSON, and images for scalable machine learning data ingestion and export.
ovachieverProcess distributed ML data pipelines with Ray Data across CPU/GPU clusters.
tadod12Orchestrate distributed ML training across multi-node clusters with Ray Train.
ovachieverOrchestrate distributed training across clusters with Ray Train and Ray Tune.
Estrella-231Transform in-memory datasets with Polars expression-based DataFrame and LazyFrame APIs.
