Benchmarks sglang on AMD GPUs using Python scripts and parallel configurations.
npx skills add https://github.com/alexsun07/agent_skills --skill sglang-amd-bench
Please help me install this Agent Skill. Skill: sglang-amd-bench Source: https://github.com/alexsun07/agent_skills/tree/main/skills/sglang-amd-bench Command: npx skills add https://github.com/alexsun07/agent_skills --skill sglang-amd-bench
This Skill allows you to benchmark the performance of sglang on AMD GPUs with various parallel configurations, providing insights into throughput, latency, and configuration optimization.
Run the sglang-amd-bench skill with the following command:
sglang-amd-bench run --model /path/to/model --config DP8EP8
High-intent search queries and answers about installing and using this skill.
To benchmark sglang performance on AMD GPUs, run the benchmarking scripts with your specified model path and parallel configuration, such as TP, DP, or EP, to collect throughput and latency metrics.
Benchmarking sglang on AMD GPUs collects key performance metrics including Time To First Token (TTFT), Time Per Output Token (TPOT), overall throughput, and per-GPU throughput to evaluate serving efficiency.
Yes, you need aiter and mori dependencies alongside sglang to properly execute server setup and benchmarking scripts for evaluating deep learning models on AMD GPU hardware configurations.
Yes, you can compare different parallel configurations for sglang serving. The benchmarking process evaluates setups like TP, DP, and EP across various workload shapes including ISL, OSL, and concurrency levels to identify the most efficient configuration.
The best way to optimize sglang throughput and latency on AMD GPUs is by running comparative benchmarks across different workload shapes and parallel configurations to identify the most efficient serving setup for your specific model.
Low per-GPU throughput during sglang serving on AMD GPUs often indicates a suboptimal parallel configuration, which can be diagnosed by benchmarking different workload shapes like ISL, OSL, and concurrency to find a more efficient setup.
alexsun07Automate tuning of AITER CK GEMM and fused MoE kernels on AMD GPUs.
jpoindexterDiagnose and optimize LLM inference performance metrics like TTFT and TPOT.
yushengsu-thuCompare and profile SGLang serving configurations with automated accuracy and performance tests.
scottgl9Automate repeatable LLM benchmarking across hardware and model configurations.