skills.rest
Supporter09
Supporter09COMMUNITY
Explore all skills by this author →
GitHub

ray-data

Process large ML datasets with distributed batch operations and streaming execution.

Updated May 4, 2026
Search Tags:#etl#distributed computing#ray#data preprocessing#gpu acceleration#batch inference
One-click install
npx skills add https://github.com/Supporter09/Face_Anti_Spoofing_Biometric --skill ray-data-supporter09
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ray-data
Source: https://github.com/Supporter09/Face_Anti_Spoofing_Biometric/tree/main/.claude/skills/ray-data
Command: npx skills add https://github.com/Supporter09/Face_Anti_Spoofing_Biometric --skill ray-data-supporter09

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Ray Data removes the bottleneck of loading and transforming large datasets on a single machine, so ML teams can preprocess, infer, and train on data that no longer fits in memory.

Core Features & Use Cases

  • Distributed, streaming execution for large batch jobs and ETL pipelines.
  • Multi-modal readers and writers for Parquet, CSV, JSON, images, NumPy, and more.
  • Vectorized batch transforms, GPU-accelerated preprocessing, and direct integration with Ray Train, PyTorch, and TensorFlow.
  • Use it to build offline inference pipelines, image preprocessing workflows, and scalable training data preparation.

Quick Start

Install Ray Data, load your dataset, apply a batch transform, and iterate over the results to process or export them.

Frequently Asked Questions about ray-data

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I preprocess large machine learning datasets that no longer fit in memory?▼

Distributed data preprocessing removes the single-machine memory bottleneck by streaming large datasets across a cluster. You can apply batch transforms to process data incrementally, keeping memory usage bounded regardless of total dataset size.

Can I run batch inference and ETL pipelines on multi-modal data formats like Parquet and images?▼

Batch inference and ETL pipelines support multi-modal readers and writers for Parquet, CSV, JSON, images, NumPy, audio, and video. You apply vectorized map_batches transforms to execute these operations in distributed, streaming batches.

Does distributed data preprocessing work with PyTorch and TensorFlow training workflows?▼

Distributed data preprocessing integrates directly with Ray Train, PyTorch, and TensorFlow. You can prepare scalable training data and feed it into your ML training workflows without leaving the distributed computing environment.

How do I use GPU acceleration for image preprocessing workflows?▼

GPU acceleration for image preprocessing is handled through vectorized batch transforms. By applying map_batches operations, you execute GPU-accelerated transformations across your dataset in a distributed manner.

What is the best way to process huge ML datasets across single-node and cluster deployments?▼

Streaming execution enables processing huge ML datasets across both single-node and cluster deployments. This approach uses distributed batch operations to handle data loading, preprocessing, and inference at scale without memory overflow.

Related Skills

Explore More Data & Analytics Skills→
ricablericable

ray-data

Streams distributed, batched ML data preprocessing across clusters for PyTorch and TensorFlow workflows.

Community
Advanced
tadod12tadod12

ray-train

Orchestrate distributed ML training across multi-node clusters with Ray Train.

Community
Advanced
qcmuuqcmuu

ray-data

Scale ML data preprocessing and batch inference with Ray Data across distributed workers.

Community
Advanced
cas-bigdatalabcas-bigdatalab
541

ray_image_deduplicator

Eliminate duplicate images in large datasets using Ray distributed computing.

Official
Advanced