skills.rest
saintgo7
saintgo7COMMUNITY
Explore all skills by this author →
GitHub

gem-llm-load-test

Generate concurrent load tests for GEM-LLM and collect latency and throughput metrics.

Updated Apr 4, 2026
Search Tags:#rate limiting#load testing#throughput#locust#gem-llm#latency p99#asyncio benchmarking
One-click install
npx skills add https://github.com/saintgo7/claude-skills --skill gem-llm-load-test
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gem-llm-load-test
Source: https://github.com/saintgo7/claude-skills/tree/main/gem-llm-load-test
Command: npx skills add https://github.com/saintgo7/claude-skills --skill gem-llm-load-test

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill eliminates uncertainty in GEM-LLM performance by running repeatable load tests that expose real throughput, latency (including p99), and quota/rate-limit behaviors under concurrent users.

Core Features & Use Cases

  • Concurrent load generation (locust + asyncio bench): Runs both a single-key Locust workload and a multi-key multi-user asyncio benchmark to measure realistic behavior.
  • Quota and rate-limit scenario validation: Confirms 60RPM and daily token limits via controlled concurrency and user-key setup.
  • Production-oriented metrics capture: Produces latency percentiles (p50/p95/p99/max), success/failure classifications (HTTP 401/429/500), and token throughput (tok/s), saving results into reports for analysis.

Use Case

Validate whether a new deployment can sustain a target concurrency (e.g., ~50 concurrent requests), keeps p99 latency under a threshold (e.g., <5s), and meets throughput requirements (e.g., >30 req/s), while ensuring quota/rate-limit responses behave correctly.

Quick Start

Run the skill with install.sh for gem-llm-load-test, then execute the Locust single-key test or the multi-user-bench script to generate load and save metrics reports.

Frequently Asked Questions about gem-llm-load-test

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I load test LLM throughput and p99 latency under concurrent traffic?▼

To load test LLM throughput and p99 latency, you generate controlled concurrent traffic using locust and asyncio runners to collect realistic latency percentiles and throughput results. This validates performance and quota behaviors under concurrent user scenarios.

What is the best way to verify API rate limiting and 429 quota responses for LLMs?▼

Verifying API rate limiting and 429 quota responses for LLMs requires running controlled concurrency tests via locust and multi-user asyncio benchmarks. This approach validates 60RPM and daily token limits while classifying HTTP 401, 429, and 500 success and failure rates.

Can I benchmark LLM scalability using both single-key and multi-key setups?▼

You can benchmark LLM scalability using both single-key and multi-key setups by running a single-key locust workload and a multi-user asyncio benchmark. This combination measures realistic behavior across short chat, coding help, long context, and streaming multi-tool workloads.

How does asyncio benchmarking capture p50 and p99 latency metrics for LLM endpoints?▼

Asyncio benchmarking captures p50 and p99 latency metrics by generating concurrent LLM endpoint traffic and calculating latency percentiles including p50, p95, p99, and max. It also records token throughput in tok/s and saves these production-oriented metrics into analysis reports.

Do I need locust to validate GEM-LLM concurrency and success rate thresholds?▼

You need locust and the multi-user-bench asyncio runner to validate GEM-LLM concurrency and success rate thresholds. These tools generate the controlled load required to confirm target concurrency, p99 latency boundaries, and throughput requirements while ensuring correct rate-limit responses.

Why does LLM p99 latency spike under high concurrency during load testing?▼

LLM p99 latency spikes under high concurrency during load testing when rate limits or quota thresholds are exceeded, triggering HTTP 429 responses. Load testing exposes these scalability bottlenecks by classifying HTTP errors and measuring latency degradation under concurrent user pressure.

Related Skills

Explore More Data & Analytics Skills→
saintgo7saintgo7

concurrent-load-testing-pattern

Measure throughput and p50/p95/p99 latency for concurrent LLM API requests.

Community
Intermediate
adaptive-enforcement-labadaptive-enforcement-lab
3

error-handling

Detect and retry GitHub Actions token, permission, rate limit, and network failures.

Official
Advanced
nntan90nntan90
4

performance-test

Automate end-to-end performance testing for APIs and services with k6, Locust, and JMeter.

Community
Advanced
saintgo7saintgo7

llm-serving-performance-tuning

Diagnose and resolve LLM serving bottlenecks across vLLM, FastAPI, and database.

Community
Advanced