skills.rest
box755
box755COMMUNITY
Explore all skills by this author →
GitHub

ray-data

Stream dataset reading and distributed transformations for scalable ML data processing.

Updated Aug 23, 2026
Search Tags:#parquet#ray data#batch inference#ml pipelines#ray train#gpu preprocessing#distributed processing
One-click install
npx skills add https://github.com/box755/simlens-research --skill ray-data-box755
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ray-data
Source: https://github.com/box755/simlens-research/tree/main/skill-packs/AI-Research-SKILLs/05-data-processing/ray-data
Command: npx skills add https://github.com/box755/simlens-research --skill ray-data-box755

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires ray[data], pyarrow, pandas, and includes references (resource) components.

What problem does it solve?

Ray Data helps you process large datasets efficiently for machine learning by providing scalable, streaming data transformations that can run across CPUs/GPUs.

Core Features & Use Cases

  • Streaming execution: Process datasets larger than memory while keeping the pipeline lazy and scalable.
  • Distributed transforms: Apply batch and row operations (map, filter, groupby/aggregate) in parallel across a cluster.
  • Multi-modal & format support: Load and transform Parquet/CSV/JSON and images for common ML and ETL workflows.
  • Framework integration: Work with Ray Train to shard datasets and convert results to PyTorch or TensorFlow-friendly formats.
  • Use cases: Batch inference, distributed data preprocessing, multi-modal data loading, and distributed ETL pipelines.

Quick Start

Use the ray-data skill to load Parquet data, apply a batch transformation, and iterate over produced batches for downstream ML steps.

Frequently Asked Questions about ray-data

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run distributed ML data preprocessing on datasets larger than memory?▼

Distributed batch inference applies map_batches transformations in parallel across a cluster, allowing you to process large datasets and generate predictions efficiently without loading everything into memory.

Can I use GPU acceleration for batch transformations in Ray Data pipelines?▼

Yes, GPU-accelerated map_batches transformations are supported for batch computation. This allows you to execute parallel data processing workloads across available GPUs in your cluster.

Does Ray Data work with Ray Train to shard datasets for PyTorch or TensorFlow?▼

Yes, Ray Data integrates with Ray Train to shard datasets and convert results into PyTorch or TensorFlow-friendly formats. This enables seamless distributed model training across multi-node clusters.

What file formats are supported for distributed ETL and multi-modal data loading?▼

Supported file formats include Parquet, CSV, JSON, and images for distributed ETL and multi-modal data loading. You can load and transform these formats using streaming dataset reading.

What is the best way to scale ML pipelines from a single laptop to a multi-node cluster?▼

Scaling ML pipelines from a laptop to a multi-node cluster involves using streaming transforms and distributed processing. This approach allows you to develop locally and scale to a cluster without changing code.

Related Skills

Explore More Data & Analytics Skills→
hhhi21ghhhi21g

ray-data

Process large datasets for ML with distributed computing across CPU/GPU environments.

Community
Advanced
qcmuuqcmuu

ray-data

Scale ML data preprocessing and batch inference with Ray Data across distributed workers.

Community
Advanced
OpenLAIROpenLAIR
1.0k

ray-train

Orchestrate distributed machine learning training across multiple nodes and clusters.

Official
Advanced
Hung-3008Hung-3008
1

dask

Distributes pandas/NumPy workloads across cores and machines for large-scale data processing.

Community
Advanced