Distribute and accelerate ML data processing with Ray Data.
npx skills add https://github.com/arsity/scholar-tools --skill ray-data-arsity
Please help me install this Agent Skill. Skill: ray-data Source: https://github.com/arsity/scholar-tools/tree/main/vendor/ai-research-skills/05-data-processing/ray-data Command: npx skills add https://github.com/arsity/scholar-tools --skill ray-data-arsity
Ray Data provides scalable distributed data processing for ML workflows, enabling faster preprocessing and data preparation on large datasets.
Process a small dataset with a simple Ray Data pipeline to read Parquet files, apply a transformation, and write the results back to Parquet.
High-intent search queries and answers about installing and using this skill.
Distributed data processing for ML pipelines scales by streaming execution across CPU and GPU clusters, enabling faster preprocessing and data preparation on large datasets without being limited by single machine memory.
Ray Data integrates directly with PyTorch, TensorFlow, and HuggingFace frameworks, allowing you to build end-to-end ML pipelines that load and transform distributed datasets natively within your existing training loops.
Multi-node ETL on Parquet and CSV files is best handled by streaming execution pipelines that read standard data formats, apply distributed transformations across clusters, and write results back to Parquet for scalable batch processing.
Batch preprocessing pipelines for large ML datasets are set up by reading data formats like Parquet or JSON, applying transformations across a distributed cluster, and writing the processed results back to disk using streaming execution.
Distributed ETL supports streaming data workflows and processes multiple data formats including Parquet, CSV, JSON, and images, making it suitable for multi-modal data loading and continuous streaming execution across clusters.
Distributed streaming execution for ML data preparation is not necessary for small datasets that fit into single-machine memory, as the overhead of cluster coordination outweighs the performance benefits of distributed processing.
hhhi21gProcess large datasets for ML with distributed computing across CPU/GPU environments.
tadod12Orchestrate distributed ML training across multi-node clusters with Ray Train.
TeoSlayerDeploy a four-agent ML training pipeline coordinating data preparation, training, evaluation, and serving.
manutejTeach Apache Spark data processing across RDDs, DataFrames, Spark SQL, streaming, and MLlib.
