skills.rest
handsomelong922
handsomelong922COMMUNITY
Explore all skills by this author →
GitHub

ray-data

Scale distributed data processing for ML pipelines across clusters.

Updated Apr 3, 2026
Search Tags:#distributed#transformation#ray#parquet#dataset#data-processing#ml-pipelines
One-click install
npx skills add https://github.com/handsomelong922/my-codex-skills --skill ray-data-handsomelong922
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ray-data
Source: https://github.com/handsomelong922/my-codex-skills/tree/main/skills/ray-data
Command: npx skills add https://github.com/handsomelong922/my-codex-skills --skill ray-data-handsomelong922

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Ray Data provides scalable, distributed data processing for machine learning workloads, enabling teams to preprocess, augment, and feed large datasets without memory constraints or manual orchestration.

Core Features & Use Cases

  • Streaming and bulk data processing that scales from a laptop to a multi-node cluster.
  • Framework integration with PyTorch, TensorFlow, and Ray Train for end-to-end ML pipelines.
  • Support for common data formats (Parquet, CSV, JSON, images) and seamless data ingestion, transformation, and write-back.
  • Use cases include preparing training data, batch inference pipelines, and multi-modal data loading for ML experiments.

Quick Start

Install Ray Data and run a simple data pipeline that reads a dataset, applies a transformation, and writes the result.

Frequently Asked Questions about ray-data

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scale distributed data processing for ML pipelines across a cluster?▼

Distributed data processing for ML pipelines scales from a laptop to a multi-node cluster by streaming large datasets, enabling ingestion, transformation, and loading for training and inference without memory constraints.

Can I use Ray Data with PyTorch and TensorFlow for machine learning pipelines?▼

Yes, distributed ML data processing integrates directly with PyTorch, TensorFlow, and Ray Train, allowing you to feed transformed data into end-to-end training and evaluation pipelines.

What data formats are supported for distributed transformation in ML pipelines?▼

Distributed data transformation supports common ML data formats including Parquet, CSV, JSON, and images, enabling seamless ingestion, transformation, and write-back for large datasets.

How does streaming execution handle large datasets for batch inference?▼

Streaming execution processes large datasets for batch inference without memory constraints by incrementally loading and transforming data across cloud or cluster environments.

Does distributed ML data processing support GPU across multiple nodes?▼

Yes, distributed ML data processing provides seamless GPU support across multiple nodes, enabling accelerated transformation and loading for large-scale training workloads.

What is the best way to prepare multi-modal training data without hitting memory limits?▼

Preparing multi-modal training data without memory limits requires streaming and bulk data processing that scales across a cluster, handling ingestion and augmentation for large datasets seamlessly.

Related Skills

Explore More Data & Analytics Skills→
jluo41jluo41

haipipe-data-1-source

Convert CSV, XML, Parquet, and JSON files into standardized SourceSet assets.

Community
Advanced
ovachieverovachiever
52

ray-data

Process distributed ML data pipelines with Ray Data across CPU/GPU clusters.

Community
Advanced
tadod12tadod12

ray-train

Orchestrate distributed ML training across multi-node clusters with Ray Train.

Community
Advanced
Tnemo65Tnemo65

ray-data

Scale distributed data processing for ML workloads with streaming and batch pipelines.

Community
Advanced