ray

Distributed processing with Ray converts functions into parallelizable tasks using ray.remote, enabling concurrent execution across CPU and GPU clusters. Ray handles resource management and fault tolerance automatically, scaling from single machines to large clusters without code changes.

Ray Data pipelines enable batch processing of thousands of files concurrently. Use map, map_batches, and batch writes to orchestrate end-to-end ETL workflows, with built-in parallelization across available resources for video processing, PDF parsing, and multi-step data operations.

Ray Tasks parallelize stateless functions for concurrent execution, while Ray Actors maintain loaded models and state across requests, avoiding repeated initializations. Choose Tasks for one-off parallel work and Actors for services requiring persistent state or resource reuse.

GPU support is optional in Ray. The core library runs on CPU for standard batch processing and ETL tasks. GPU acceleration is available for compute-intensive workloads like video processing when optional GPU components are configured in your cluster.

Ray provides fault-tolerant orchestration through distributed task scheduling and automatic recovery. Failed tasks can be retried, and Ray's lineage tracking enables recovery without reprocessing completed steps in multi-step pipelines across clusters.