ray-data

Process large datasets with distributed streaming across CPU and GPU resources.

4|1|Updated Jan 7, 2026

Search Tags:#mlops #data processing #etl #distributed computing #data pipelines #ray

Ready to run

npx skills add https://github.com/an8079/take-skills --skill ray-data-an8079

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: ray-data
Source: https://github.com/an8079/take-skills/tree/main/skills/ray-data
Command: npx skills add https://github.com/an8079/take-skills --skill ray-data-an8079

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires ray[data], pyarrow, pandas, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the challenge of processing large datasets for machine learning workloads, enabling efficient data manipulation and preparation at scale.

Core Features & Use Cases

Distributed Data Processing: Handles datasets larger than memory using streaming execution across CPU/GPU.
Multi-Format Support: Ingests and processes data from various formats like Parquet, CSV, JSON, and images.
Framework Integration: Seamlessly integrates with ML frameworks such as PyTorch, TensorFlow, and Ray Train.
Use Case: Use this Skill to preprocess terabytes of image data for a computer vision model, distributing the workload across a cluster of machines for faster training.

Quick Start

Use the ray-data skill to read all parquet files from 's3://my-bucket/data/' and transform the 'text' column to lowercase.

Frequently Asked Questions about ray-data

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I process datasets larger than memory for machine learning?▼

You can process datasets larger than memory using streaming execution across CPU and GPU resources. This approach enables scalable data manipulation for machine learning workloads without loading everything into RAM.

Can I use this for distributed ETL pipelines with Parquet and CSV files?▼

Yes, distributed ETL pipelines support ingesting and processing multiple data formats including Parquet, CSV, JSON, and images. You can read from storage like S3 and apply transformations across a cluster of machines.

Does distributed data processing work with PyTorch and TensorFlow?▼

Yes, distributed data processing integrates seamlessly with ML frameworks such as PyTorch, TensorFlow, and Ray Train. You can feed preprocessed data directly into distributed model training workflows.

What is the best way to scale batch inference across a cluster?▼

Batch inference optimization scales workloads across a cluster of machines for faster execution. Distributed computing assigns inference tasks across available CPU and GPU resources to process large data volumes efficiently.

Do I need Ray and PyArrow to run distributed ETL pipelines?▼

Yes, dependencies include ray[data], pyarrow, and pandas to execute distributed ETL pipelines. These libraries provide the underlying execution engine and data structure support required for scalable processing.