modal-serverless-gpu

To deploy a serverless GPU API, you define a Modal app that configures a containerized image, selects a GPU like T4 or A10G, and exposes model inference through a FastAPI or ASGI web endpoint. This enables on-demand execution with automatic scaling to zero when idle.

Running scheduled batch inference on serverless GPUs involves configuring Modal functions with cron-like scheduling to process datasets or run periodic evaluations. This approach minimizes idle infrastructure costs by automatically scaling compute resources down to zero between scheduled executions.

Yes, you can use FastAPI with Modal for ML deployment by applying ASGI endpoint decorators to your Python functions. This allows you to serve model predictions directly as REST APIs while Modal handles the underlying serverless GPU compute and autoscaling.

To keep model weights cached for serverless inference and reduce cold starts, you configure Modal volumes for persistent storage within your Python-native infrastructure. This allows container warm-ups and dynamic batching to maintain ready states across function invocations.

You should select a GPU based on your workload requirements: choose a T4 for quick prototypes, or use L40S and A10G GPUs for common inference tasks. Modal allows explicit resource specification to match your specific machine learning compute needs.

Yes, serverless GPU compute supports training and fine-tuning models on-demand. By defining scalable Python functions within Modal, you can execute fine-tuning workloads while minimizing idle infrastructure costs through automatic scaling to zero when the process completes.