skills.rest
SerendipityOneInc
SerendipityOneIncOFFICIAL
Explore all skills by this author →
GitHub

raydata

Automate Ray Data job development, testing, deployment, and Airflow scheduling.

Updated Aug 27, 2026
Search Tags:#workflow#production#GPU#data-processing#ray-data#distributed-computing
One-click install
npx skills add https://github.com/SerendipityOneInc/srp-claude-code-marketplace --skill raydata
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: raydata
Source: https://github.com/SerendipityOneInc/srp-claude-code-marketplace/tree/main/plugins/srp-developer/skills/raydata
Command: npx skills add https://github.com/SerendipityOneInc/srp-claude-code-marketplace --skill raydata

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Automates the development lifecycle of Ray Data jobs.

Core Features & Use Cases

  • Local development on A10 with remote SSH, virtual environments, and code storage at /data/srp/raydata/code/.
  • Slurm-based testing on H100/H200 with apptainer for GPU/CPU coordination.
  • Production deployment on RayCluster with runtime-env configuration and ray job submission.
  • Scheduling and orchestration with Airflow for periodic runs.
  • Example workflows include image classification and vision-language pipelines using Ray Data.

Quick Start

Connect to an A10 development machine using SSH. Create a virtual environment: python3 -m venv my_venv; source my_venv/bin/activate Install dependencies: pip install ray[data] transformers Pillow torch Place your code under /data/srp/raydata/code/ Run a local test: python /data/srp/raydata/code/your-job.py

Frequently Asked Questions about raydata

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I develop and deploy Ray Data jobs for distributed GPU processing?▼

Ray Data automates the full development lifecycle: local testing on A10 GPUs, validation on H100/H200 via Slurm, and production deployment to RayCluster. It manages runtime environments, dependencies, and job submission end-to-end.

Can I test Ray Data workflows on multiple GPU types before production?▼

Yes. Ray Data supports local A10 development, then Slurm-based testing on H100/H200 GPUs with Apptainer containerization for isolated CPU/GPU coordination before RayCluster deployment.

How do I schedule Ray Data jobs to run periodically in production?▼

Ray Data integrates with Airflow for orchestration and scheduling. After deploying to RayCluster, you configure Airflow workflows to trigger periodic Ray job submissions with managed runtime environments.

What's required to set up a Ray Data workflow locally?▼

Create a Python virtual environment, install ray[data] with transformers and torch, store code in /data/srp/raydata/code/, then run jobs directly. SSH access to an A10 machine is the primary prerequisite.

Does Ray Data handle dependency management across development and production environments?▼

Yes. Ray Data enforces runtime-env configuration to ensure consistent Python packages, versions, and dependencies between local virtual environments, Slurm tests, and RayCluster deployments.

What GPU-accelerated workflows does Ray Data support?▼

Ray Data includes example implementations for image classification and vision-language model pipelines, demonstrating how to parallelize data processing across distributed GPUs for production scale.

Related Skills

Explore More Data & Analytics Skills→
Orchestra-ResearchOrchestra-Research
11.5k

ray-data

Process large datasets with distributed transformations across CPU and GPU resources.

Official
Intermediate
ivanshamaevivanshamaev
14

dataops-airflow-observability

Export Apache Airflow scheduler, task, and DAG signals to metrics, traces, and logs.

Community
Advanced
NVIDIA-TAONVIDIA-TAO
83

tao-run-on-slurm

Automate remote TAO GPU job submission on SLURM clusters via SSH.

Official
Advanced
SalesforceCommerceCloudSalesforceCommerceCloud
51

b2c-ods

Provision, manage, and monitor on-demand B2C sandboxes using the b2c CLI.

Official
Intermediate