ray-data

Process large-scale ML data with Ray Data for distributed preprocessing and ETL.

1.0k|117|Updated Feb 26, 2026

Search Tags:#distributed #ray #parquet #data-processing #ray-data #ml-pipelines #data-preprocessing

Ready to run

npx skills add https://github.com/OpenLAIR/dr-claw --skill ray-data-openlair

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: ray-data
Source: https://github.com/OpenLAIR/dr-claw/tree/main/skills/data-processing/ray-data
Command: npx skills add https://github.com/OpenLAIR/dr-claw --skill ray-data-openlair

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Ray Data provides scalable, distributed data processing for machine learning workloads, simplifying large-scale preprocessing and ETL across clusters.

Core Features & Use Cases

Streaming and batch data processing across Parquet/CSV/JSON/images
Framework integration with PyTorch, TensorFlow, and HuggingFace
End-to-end data pipelines for batch inference and preprocessing on large datasets

Quick Start

Enable scalable data workflows by installing Ray Data and running a sample preprocessing pipeline.

Frequently Asked Questions about ray-data

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I scale ML data preprocessing across large Parquet and CSV datasets?▼

You scale ML data preprocessing across large Parquet and CSV datasets by applying distributed streaming execution across clusters. This approach handles batch and ETL transformations efficiently for machine learning pipelines.

Can I process JSON and image data for batch inference pipelines?▼

Yes, you can process JSON and image files for batch inference pipelines. The distributed data processing library supports multi-format streaming transformations to enable end-to-end preprocessing on large datasets.

Does distributed data processing work with PyTorch and TensorFlow frameworks?▼

Distributed data processing integrates directly with PyTorch, TensorFlow, and HuggingFace frameworks. This integration allows seamless feeding of preprocessed data into machine learning pipelines for training and inference.

What is the best way to run ETL transformations on large-scale ML data?▼

The best way to run ETL transformations on large-scale ML data is utilizing distributed streaming execution with GPU acceleration. This method handles multi-format data processing efficiently across computing clusters.

When should I use distributed streaming execution for data preprocessing?▼

Use distributed streaming execution for data preprocessing when handling large-scale ML workloads that exceed single-node memory capacity. It enables scalable ETL across clusters, supporting GPU acceleration for multi-format datasets.