modal-serverless-gpu

Define a Modal App with a function or class specifying a GPU and container image, then attach @modal.fastapi_endpoint() or @modal.asgi_app() to expose it over HTTP. Run modal deploy script.py to get a persistent HTTPS endpoint with zero-downtime updates.

L40S with 48GB VRAM offers the best cost-to-performance ratio for inference workloads. T4 suits budget inference of small models, while A100-80GB and H100 are better for very large models or training.

Modal fits serverless, scale-to-zero workloads with pay-per-second billing and Python-defined infrastructure. RunPod suits longer-running pods with persistent state, while SkyPilot handles multi-cloud orchestration and cost optimization across providers.

Increase container_idle_timeout to keep containers warm, use keep_warm=1 for always-ready instances, and load models inside @modal.enter() so initialization happens once at container start rather than per request. Caching models in a Volume also avoids repeated downloads.

CUDA OOM occurs when the model exceeds GPU VRAM. Switch to a larger GPU like A100-80GB, enable gradient checkpointing, use mixed precision with torch.autocast, or enable flash attention to reduce memory consumption.

Use modal.Volume.from_name with create_if_missing=True and mount it into your function. After writing files, call volume.commit() to persist changes, and call volume.reload() in readers to fetch the latest data.