skills.rest
informatico-madrid
informatico-madridCOMMUNITY
Explore all skills by this author →
GitHub

ray-data

Process large datasets with Ray Data for ML preprocessing and ETL.

1|Updated Feb 10, 2026
Search Tags:#data processing#etl#distributed computing#ray data#scalable#ml pipelines
One-click install
npx skills add https://github.com/informatico-madrid/Architect-Expert-Gap-Forge --skill ray-data-informatico-madrid
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ray-data
Source: https://github.com/informatico-madrid/Architect-Expert-Gap-Forge/tree/main/.github/skills/ray-data
Command: npx skills add https://github.com/informatico-madrid/Architect-Expert-Gap-Forge --skill ray-data-informatico-madrid

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires ray[data], pyarrow, pandas, and includes references (resource) components.

What problem does it solve?

This Skill addresses the challenge of processing large datasets efficiently for Machine Learning workloads, enabling distributed data preprocessing and scalable data loading.

Core Features & Use Cases

  • Distributed Data Processing: Handles datasets larger than memory using streaming execution across multiple nodes.
  • Multi-modal Data Support: Reads and processes various data formats including Parquet, CSV, JSON, and images.
  • Framework Integration: Seamlessly integrates with ML frameworks like PyTorch and TensorFlow, and with Ray Train for distributed training.
  • Use Case: Preprocess terabytes of image data for a computer vision model, distributing the transformations across a cluster of machines for faster iteration.

Quick Start

Use the ray-data skill to read all parquet files from 's3://my-bucket/data/' and apply a lowercase transformation to the 'text' column.

Frequently Asked Questions about ray-data

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I process datasets larger than memory for machine learning pipelines?▼

Distributed data processing handles datasets larger than memory by using streaming execution across multiple nodes, enabling scalable ETL and data preprocessing for ML pipelines without loading everything into RAM.

Can I use Ray Data with PyTorch and TensorFlow for distributed training?▼

Yes, Ray Data integrates seamlessly with ML frameworks like PyTorch and TensorFlow, and works with Ray Train to provide scalable data loading for distributed training workloads.

What data formats are supported for distributed ETL and preprocessing?▼

Supported data formats include Parquet, CSV, JSON, and images, allowing you to read and process diverse multi-modal data types for batch inference and ETL pipelines.

Does scalable ML data processing support GPU acceleration?▼

Yes, scalable ML data processing supports GPU acceleration to speed up transformations, making it suitable for preprocessing terabytes of image data for computer vision models.

What's the best way to preprocess terabytes of image data across a cluster?▼

Distributed data processing distributes transformations across a cluster of machines using streaming execution, allowing faster iteration when preprocessing terabytes of image data for ML models.

Related Skills

Explore More Software Engineering Skills→
hhhi21ghhhi21g

ray-data

Process large datasets for ML with distributed computing across CPU/GPU environments.

Community
Advanced
tadod12tadod12

ray-train

Orchestrate distributed ML training across multi-node clusters with Ray Train.

Community
Advanced
ihatesea69ihatesea69
3

ray-train

Orchestrate distributed ML training across multi-node clusters with Ray Train.

Community
Advanced
galeepgaleep

dask

Scale pandas and NumPy data processing beyond memory with Dask.

Community
Advanced