skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

ray-data

Process large-scale ML datasets across clusters for distributed preprocessing and batch inference.

2|Updated Apr 12, 2026
Search Tags:#distributed#gpu#etl#ray#parquet#data-processing#batch-inference
One-click install
npx skills add https://github.com/Clay-HHK/claude-config --skill ray-data-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ray-data
Source: https://github.com/Clay-HHK/claude-config/tree/main/skills/AI-research-SKILLs/05-data-processing/ray-data
Command: npx skills add https://github.com/Clay-HHK/claude-config --skill ray-data-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Ray Data simplifies processing, transforming, and streaming very large machine learning datasets that exceed single-machine memory, removing the friction of distributed preprocessing and batch inference so teams can move from local experiments to cluster-scale production pipelines.

Core Features & Use Cases

  • Streaming execution for datasets larger than memory, enabling iterative processing with bounded resources.
  • GPU-accelerated transforms to speed up image/audio preprocessing and other heavy compute steps.
  • Multi-format ingestion and output including Parquet, CSV, JSON, images, NumPy, and Pandas interoperability for ETL and model training workflows.
  • Framework integration with Ray Train, PyTorch, and TensorFlow for seamless data->training pipelines and distributed batch inference.
  • Use case: Run distributed preprocessing and feature engineering over hundreds of gigabytes stored on S3, then feed the processed Parquet shards into a Ray Train job for scalable model training.

Quick Start

Load a Parquet dataset from s3://bucket/data/*.parquet and run a batched lowercase text transform with a batch size tuned for your cluster.

Frequently Asked Questions about ray-data

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I load and transform a Parquet dataset for a distributed training job?▼

You can load a Parquet dataset from local filesystems or cloud storage and run batched transforms with a tuned batch size. The processed data is then output as Parquet shards, ready to be consumed by distributed training jobs.

How do I run distributed preprocessing on large ML datasets stored in S3?▼

Distributed preprocessing on S3 is handled by streaming execution, which processes datasets larger than memory across clusters. You can ingest Parquet or CSV files directly from S3 and apply configurable parallelism to transform hundreds of gigabytes of data.

Can I use GPU-accelerated transforms for image preprocessing in a batch inference pipeline?▼

GPU-accelerated transforms are supported to speed up image and audio preprocessing for batch inference. This allows heavy compute steps to be distributed across cluster resources, enabling multi-modal data loading and efficient inference workflows.

What is the best way to process datasets larger than memory for ETL pipelines?▼

Streaming execution enables ETL pipelines to process datasets larger than memory by iterating with bounded cluster resources. It supports multi-format ingestion including Parquet, CSV, and JSON, allowing continuous data transformation without loading everything at once.

Does this distributed data processing work with PyTorch and TensorFlow for model training?▼

Framework integration with PyTorch and TensorFlow is fully supported for seamless data-to-training pipelines. You can feed processed Parquet shards directly into Ray Train jobs, enabling scalable model training and distributed batch inference.

How do I load and transform a Parquet dataset for a distributed training job?▼

You can load a Parquet dataset from local filesystems or cloud storage and run batched transforms with a tuned batch size. The processed data is then output as Parquet shards, ready to be consumed by distributed training jobs.

Related Skills

Explore More Data & Analytics Skills→
AstorYHAstorYH
15

snowflake-table-export

Export Snowflake table data to CSV, JSON, or Parquet files.

Community
Intermediate
informatico-madridinformatico-madrid
1

ray-data

Process large datasets with Ray Data for ML preprocessing and ETL.

Community
Advanced
ovachieverovachiever
52

ray-data

Process distributed ML data pipelines with Ray Data across CPU/GPU clusters.

Community
Advanced
viniruggeriviniruggeri

dask

Distribute large pandas and NumPy workloads across clusters for parallel processing.

Community
Advanced