perf-test-flagos

To benchmark vLLM model performance, you must start a running vLLM server with your target model, then execute the included run_benchmark scripts to measure latency, throughput, TTFT, and TPOT across five workload profiles.

Measuring AI model speed uses five workload profiles consisting of short and long prefill combined with short and long decode, plus high concurrency, to capture comprehensive latency, throughput, TTFT, and TPOT metrics.

Yes, you need a running vLLM server and a target model path to test AI model latency and TTFT, as the benchmarking scripts orchestrate tests directly against the deployed vLLM instance.

You can run accuracy benchmarks alongside performance tests for vLLM models by utilizing FlagEval for accuracy evaluation while simultaneously using vLLM bench serve for performance measurements.

The vLLM benchmark report includes throughput, latency, Time To First Token (TTFT), and Time Per Output Token (TPOT) metrics, evaluating the model across five distinct workload profiles to generate a combined performance report.