modal-serverless-gpu

Define a modal.App, attach a function with a GPU and container image, then add the @modal.fastapi_endpoint or @modal.asgi_app decorator. Run modal deploy script.py to get a persistent HTTPS endpoint with automatic scaling.

L40S with 48GB VRAM offers the best cost-to-performance ratio for inference. T4 suits budget workloads with small models, while A100-80GB and H100 handle very large models requiring more memory.

Modal fits serverless, scale-to-zero workloads with Python-defined infrastructure. RunPod suits longer-running pods with persistent state, Lambda Labs offers reserved instances, and SkyPilot handles multi-cloud orchestration and cost optimization.

Increase container_idle_timeout to keep containers warm, use keep_warm=1 for always-ready instances, and load models inside an @modal.enter() lifecycle hook so initialization happens once at container start rather than per request.

CUDA OOM occurs when the model exceeds GPU VRAM. Switch to a larger GPU like A100-80GB, enable gradient checkpointing, use mixed precision with torch.autocast, or reduce batch sizes to fit within available memory.

Use modal.Volume.from_name with create_if_missing=True and mount it into your function. Save models to the mounted path and call volume.commit() to persist changes, then volume.reload() to read the latest data.