ray-data

Load data with ray.data.read_parquet or read_csv, apply transformations with map_batches for vectorized operations, and consume results with iter_batches. Streaming execution lets you process data larger than available memory.

Use Pandas for small data under 1GB on a single machine, Spark for enterprise SQL-based ETL, and Ray Data for ML workloads over 100GB needing GPU acceleration and integration with training frameworks like PyTorch.

Yes, pass num_gpus=1 to map_batches to run transforms on GPUs. This is commonly used for image and audio preprocessing, where benchmarks show roughly 5x throughput per GPU compared to CPU-only execution.

Convert the dataset with ds.to_torch specifying the label column and batch size, or use iter_torch_batches. With Ray Train, pass datasets to TorchTrainer and access shards via ray.train.get_dataset_shard inside the training function.

Ray Data reads and writes Parquet, CSV, JSON, and NumPy, and reads images and pandas DataFrames. Parquet is the recommended format for ML data due to efficient columnar storage and streaming reads.

Avoid Ray Data for small datasets under 1GB where Pandas is simpler, or for SQL-heavy enterprise ETL where Spark fits better. Ray Data targets ML pipelines, batch inference, and multi-modal loading rather than general analytics queries.