llm-serving-performance-tuning

Reducing p99 latency in a vLLM serving stack requires baseline load testing metrics to identify bottlenecks across vLLM, the FastAPI gateway, and the database, followed by tiered parameter tuning and re-measurement under multi-user concurrency.

Throughput drops during concurrency scaling often happen because of bottlenecks like FastAPI gateway queue backlogs, 429 rate limiting, or database pool exhaustion, which you can diagnose by mapping 5xx distributions and GPU utilization during load testing.

Tuning vLLM and FastAPI for maximum throughput involves a 6-step workflow: gathering baseline metrics like RPS and tokens/sec, classifying bottlenecks, adjusting vLLM parameters, tuning gateway connection limits, optimizing database writes, and re-measuring.

Before tuning LLM serving performance, you need baseline load testing metrics including RPS, latency percentiles, tokens/sec, 5xx and 429 error rates, GPU utilization, and database pool status to avoid guessing bottlenecks based on GPU usage alone.

Yes, this workflow specifically applies to an LLM serving stack composed of vLLM, a FastAPI gateway, and a database, guiding tuning decisions for gateway connection limits, rate limiting, and DB write behavior under multi-user concurrency.

When GPU utilization is low but p99 latency is high, the bottleneck likely sits in the FastAPI gateway or database pool, requiring you to map symptoms like queue backlog and DB pool exhaustion to gateway and DB tuning steps.