ray-data

Distribute and accelerate ML data processing with Ray Data.

Updated Mar 16, 2026

Search Tags:#data #distributed #pipeline #etl #ray #processing #ml

Ready to run

npx skills add https://github.com/arsity/scholar-tools --skill ray-data-arsity

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: ray-data
Source: https://github.com/arsity/scholar-tools/tree/main/vendor/ai-research-skills/05-data-processing/ray-data
Command: npx skills add https://github.com/arsity/scholar-tools --skill ray-data-arsity

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Ray Data provides scalable distributed data processing for ML workflows, enabling faster preprocessing and data preparation on large datasets.

Core Features & Use Cases

Streaming execution with distributed processing across CPU/GPU clusters.
Framework integration with PyTorch, TensorFlow, and HuggingFace for end-to-end pipelines.
Support for Parquet, CSV, JSON, and images, suitable for batch and streaming ETL.
Use cases include batch preprocessing, multi-modal data loading, and distributed ETL pipelines.

Quick Start

Process a small dataset with a simple Ray Data pipeline to read Parquet files, apply a transformation, and write the results back to Parquet.

Frequently Asked Questions about ray-data

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I scale distributed data processing for ML pipelines?▼

Distributed data processing for ML pipelines scales by streaming execution across CPU and GPU clusters, enabling faster preprocessing and data preparation on large datasets without being limited by single machine memory.

Can I use Ray Data with PyTorch and TensorFlow frameworks?▼

Ray Data integrates directly with PyTorch, TensorFlow, and HuggingFace frameworks, allowing you to build end-to-end ML pipelines that load and transform distributed datasets natively within your existing training loops.

What is the best way to run multi-node ETL on Parquet and CSV files?▼

Multi-node ETL on Parquet and CSV files is best handled by streaming execution pipelines that read standard data formats, apply distributed transformations across clusters, and write results back to Parquet for scalable batch processing.

How do I set up a batch preprocessing pipeline for large ML datasets?▼

Batch preprocessing pipelines for large ML datasets are set up by reading data formats like Parquet or JSON, applying transformations across a distributed cluster, and writing the processed results back to disk using streaming execution.

Does distributed ETL support streaming data workflows and image formats?▼

Distributed ETL supports streaming data workflows and processes multiple data formats including Parquet, CSV, JSON, and images, making it suitable for multi-modal data loading and continuous streaming execution across clusters.

When should I not use distributed streaming execution for ML data preparation?▼

Distributed streaming execution for ML data preparation is not necessary for small datasets that fit into single-machine memory, as the overhead of cluster coordination outweighs the performance benefits of distributed processing.