sglang-amd-bench

What problem does it solve?

This Skill allows you to benchmark the performance of sglang on AMD GPUs with various parallel configurations, providing insights into throughput, latency, and configuration optimization.

Core Features & Use Cases

• Parallel Configurations: Benchmarks sglang serving on AMD GPUs with configurations such as TP, DP, and EP, across different workload shapes like ISL, OSL, and concurrency.
• Performance Metrics: Collects key metrics like TTFT, TPOT, throughput, and per-GPU throughput for analysis.
• Configuration Comparison: Compares the performance of different configurations to identify the most efficient setup.

Quick Start

Run the sglang-amd-bench skill with the following command:

sglang-amd-bench run --model /path/to/model --config DP8EP8