modal-serverless-gpu

To deploy a serverless GPU inference endpoint, you define a Modal app and function with explicit GPU selection, set up container image dependencies, and expose the function as a REST endpoint for serving predictions.

Yes, you can run scheduled batch jobs on serverless GPUs that scale to zero when idle. This avoids infrastructure management and idle costs by executing recurring jobs on demand using cron scheduling.

The best way to cache ML models for serverless GPU deployment is using persistent volumes. This volume-backed model caching stores downloaded frameworks like PyTorch and Transformers across executions.

Yes, Modal serverless GPU compute supports selecting specific GPU types, including memory variants. This allows cost-sensitive experimentation across different hardware configurations for multi-GPU training prototypes.

You serve a FastAPI application on a serverless GPU by configuring the Modal function as an ASGI app. You can enable dynamic batching to optimize prediction throughput while running the containerized workload.

You should avoid serverless GPUs for machine learning workloads requiring persistent state, as functions scale to zero when idle. Workloads needing continuous execution or fixed hardware clusters are better suited for dedicated infrastructure.