skills.rest
hung-phan
hung-phanCOMMUNITY
Explore all skills by this author →
GitHub

ray-data

Stream large datasets block-by-block for distributed machine learning preprocessing.

1|Updated Jun 4, 2026
Search Tags:#machine learning#data processing#distributed computing#ray#streaming data
One-click install
npx skills add https://github.com/hung-phan/ml-skills --skill ray-data-hung-phan
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ray-data
Source: https://github.com/hung-phan/ml-skills/tree/main/skills/ml-review/references/ml-libraries/ray/data
Command: npx skills add https://github.com/hung-phan/ml-skills --skill ray-data-hung-phan

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the challenge of processing datasets too large for memory by enabling streaming distributed data processing.

Core Features & Use Cases

  • Streaming Data Processing: Stream data block-by-block for efficient processing without memory overflow.
  • Flexible Transformations: Apply various transformations like tokenization, image preprocessing, and more.
  • Integration with Ray Train: Seamlessly integrate with Ray Train for distributed training.
  • Use Case: When you need to preprocess large datasets for machine learning training, such as large-scale image processing or text tokenization, and require efficient data streaming.

Quick Start

Process large CSV data for training using the ray-data skill:

/ray-data process --csv data.csv

Frequently Asked Questions about ray-data

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I process large datasets for machine learning when the data size exceeds memory limits?▼

Streamline large-scale data processing by streaming data block-by-block. This method enables efficient handling of datasets exceeding memory limits without causing memory overflow during machine learning preprocessing.

What's the best way to run distributed data preprocessing for large-scale image processing or text tokenization?▼

Apply flexible distributed transformations like image preprocessing and text tokenization using block-by-block streaming. This approach efficiently processes large datasets by streaming operations rather than loading everything into memory.

Can I use ray-data for streaming data transformations and integrate it with Ray Train?▼

Yes, ray-data supports streaming data processing and seamlessly integrates with Ray Train. This combination allows you to preprocess large datasets and directly feed them into distributed machine learning training pipelines.

Does distributed computing for streaming data support applying custom transformations to CSV files?▼

Distributed streaming supports applying custom transformations to CSV files. You can process large CSV data block-by-block to execute preprocessing tasks efficiently, preventing memory exhaustion during data loading.

When do I need block-by-block data streaming for machine learning pipelines?▼

Block-by-block data streaming is needed when preprocessing datasets for machine learning pipelines that exceed available memory. It streamlines large-scale data processing by loading and transforming data incrementally to avoid overflow.

Related Skills

Explore More Data & Analytics Skills→
hxk622hxk622
8

dask

Parallelize Python workflows and process datasets exceeding RAM with Dask.

Community
Advanced
Lord1EgyptLord1Egypt
2

deepchem

Train graph neural networks for molecular property prediction on chemical datasets.

Community
Advanced
aleph23aleph23

pytorch-lightning

Automate end-to-end deep learning workflows with PyTorch Lightning.

Community
Advanced
jonnabiojonnabio
6

ray-data

Process large datasets for distributed machine learning workloads with Ray.

Community
Advanced