deployment

To serve a quantized LLM checkpoint as an OpenAI-compatible API, deploy it using vLLM, SGLang, or TRT-LLM. The serving framework automatically reads hf_quant_config.json to apply the correct quantization flags and starts a real-time inference endpoint.

Tensor parallelism sizing for vLLM or SGLang servers is handled automatically by detecting available GPUs in your environment. The deployment workflow validates GPU availability and selects the appropriate tensor parallel size to start the inference server.

Yes, you can deploy unquantized Hugging Face checkpoints with vLLM or TRT-LLM. The deployment workflow supports both unquantized and ModelOpt-exported FP8 checkpoints, reading config.json to determine the correct serving configuration.

The best way to benchmark throughput for a ModelOpt quantized LLM is to deploy it as an OpenAI-compatible inference endpoint. By serving the checkpoint via vLLM or SGLang, you can integrate it into apps for real-time generation and throughput testing.

Yes, the LLM serving deployment workflow supports SLURM and Docker environments. It provides environment-aware operations that include optional remote deployment guidance for both SLURM and Docker contexts alongside local GPU detection.

You verify your vLLM or SGLang inference server is running correctly by checking the /health and /v1/models endpoints. The deployment workflow automatically performs these health checks after starting the server to ensure endpoint availability.