skills.rest
MesferAli
MesferAliCOMMUNITY
Explore all skills by this author →
GitHub

ray-data

Processes large-scale ML datasets with distributed streaming across CPUs and GPUs.

Updated Jan 12, 2026
Search Tags:#data processing#etl#distributed computing#ray data#scalable#ml pipelines
One-click install
npx skills add https://github.com/MesferAli/XCircle --skill ray-data-mesferali
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ray-data
Source: https://github.com/MesferAli/XCircle/tree/main/.claude/skills/ray-data
Command: npx skills add https://github.com/MesferAli/XCircle --skill ray-data-mesferali

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires ray[data], pyarrow, pandas, and includes references (resource) components.

What problem does it solve?

This Skill addresses the challenge of efficiently processing and transforming large datasets for machine learning workloads, especially when data exceeds single-machine memory or requires distributed computation.

Core Features & Use Cases

  • Scalable Data Processing: Handles datasets from gigabytes to terabytes across clusters.
  • Streaming Execution: Processes data larger than available RAM.
  • Multi-modal Support: Reads and processes various data types including Parquet, CSV, JSON, and images.
  • Framework Integration: Seamlessly integrates with Ray Train, PyTorch, and TensorFlow for end-to-end ML pipelines.
  • Use Case: Preprocess terabytes of image data for a computer vision model training job, distributing the transformations across a cluster of machines with GPU acceleration.

Quick Start

Use the ray-data skill to read all parquet files from 's3://my-data-bucket/raw/' and apply a lowercasing transformation to the 'text' column.

Frequently Asked Questions about ray-data

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I process terabytes of ML data that exceed available RAM using distributed computing?▼

Distributed computing processes terabytes of ML data exceeding RAM via streaming execution, reading formats like Parquet and CSV incrementally across clusters without loading everything into memory.

Can I use Ray Data for ETL pipelines and batch inference with PyTorch?▼

Yes, Ray Data integrates with PyTorch and TensorFlow to execute distributed ETL pipelines and batch inference. It handles data preprocessing and multi-modal data loading across CPUs and GPUs for end-to-end ML workloads.

What is the best way to scale data preprocessing for computer vision models across a cluster?▼

Scaling data preprocessing for computer vision models requires distributing image transformations across a cluster with GPU acceleration. This approach processes multi-modal image datasets concurrently, feeding preprocessed data directly into training frameworks.

Does this distributed data processing approach support reading multi-modal data formats like Parquet and images?▼

Yes, this distributed data processing approach supports reading multi-modal data formats including Parquet, CSV, JSON, and images. It natively ingests these formats to enable scalable preprocessing for machine learning pipelines.

How do I apply transformations to specific columns when reading Parquet files for ML pipelines?▼

To apply transformations to specific columns in Parquet files for ML pipelines, read the files from your storage location and map a transformation function, like lowercasing text columns, directly over the distributed dataset.

Related Skills

Explore More Data & Analytics Skills→
hhhi21ghhhi21g

ray-data

Process large datasets for ML with distributed computing across CPU/GPU environments.

Community
Advanced
ovachieverovachiever
52

ray-data

Process distributed ML data pipelines with Ray Data across CPU/GPU clusters.

Community
Advanced
qcmuuqcmuu

ray-data

Scale ML data preprocessing and batch inference with Ray Data across distributed workers.

Community
Advanced
legoutlegout
3

duckdb

Execute SQL analytical queries on local files and DataFrames.

Community
Intermediate