skills.rest
tylertitsworth
tylertitsworthCOMMUNITY
Explore all skills by this author →
GitHub

ray-data

Build scalable ML data pipelines with Ray Data streaming execution.

Updated Aug 27, 2026
Search Tags:#streaming#gpu#preprocessing#parquet#data-pipelines#ray-data
One-click install
npx skills add https://github.com/tylertitsworth/skills --skill ray-data-tylertitsworth
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ray-data
Source: https://github.com/tylertitsworth/skills/tree/main/ray-data
Command: npx skills add https://github.com/tylertitsworth/skills --skill ray-data-tylertitsworth

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Ray Data provides a scalable, streaming data processing engine to build end-to-end ML data pipelines across CPU and GPU resources, simplifying large-scale preprocessing and data transformations.

Core Features & Use Cases

  • Handling datasets from Parquet, JSON, CSV, images, and text with streaming execution
  • Building preprocessing pipelines with map, filter, flat_map, and map_batches (with UDFs)
  • GPU-accelerated preprocessing and efficient integration with Ray Train and Ray Serve

Quick Start

Create a Ray Data pipeline that reads Parquet files, applies map_batches transformations, and writes the results back to Parquet.

Frequently Asked Questions about ray-data

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build scalable ML data pipelines for preprocessing large Parquet datasets?▼

Ray Data enables scalable ML data pipelines by reading Parquet, JSON, CSV, images, and text into streaming datasets, then applying map, filter, flat_map, and map_batches transformations to preprocess large records efficiently across distributed resources.

Can I use GPU-accelerated preprocessing when streaming large datasets for machine learning?▼

GPU-accelerated preprocessing is supported via the Ray Data API, allowing you to offload intensive ML data transformations to GPU resources while maintaining streaming execution for large datasets.

What is the best way to stream preprocessed data directly into Ray Train or Ray Serve?▼

Building a Ray Data pipeline that reads from formats like Parquet or CSV and applying map, filter, and map_batches transformations allows you to stream preprocessed datasets directly into Ray Train or Ray Serve for model training or deployment.

Does Ray Data support fusing adjacent map operations to optimize data pipeline transformations?▼

Yes, Ray Data supports fusion of adjacent map operations to optimize ML data pipeline transformations, reducing overhead by combining sequential map or map_batches steps into a single execution pass during streaming.

How do I read and process images and text files for distributed ML preprocessing?▼

You can create datasets directly from images and text using the Ray Data API, then apply preprocessing functions like map or flat_map to transform the unstructured data across distributed CPU or GPU resources.

What Parquet read options are available when creating streaming datasets for ML pipelines?▼

Ray Data supports Parquet read options when creating streaming datasets, allowing you to configure how Parquet files are loaded and partitioned for ML pipelines while maintaining streaming execution to manage memory during large-scale preprocessing.

Related Skills

Explore More Data & Analytics Skills→
hhhi21ghhhi21g

ray-data

Process large datasets for ML with distributed computing across CPU/GPU environments.

Community
Advanced
ovachieverovachiever
52

ray-train

Orchestrate distributed training across clusters with Ray Train and Ray Tune.

Community
Advanced
anyscaleanyscale
2

ray

Convert Python workloads into Ray-based distributed execution plans.

Official
Advanced
OpenLAIROpenLAIR
1.0k

ray-data

Process large-scale ML data with Ray Data for distributed preprocessing and ETL.

Official
Advanced