concurrent-load-testing-pattern

What problem does it solve?

Concurrency load testing for APIs/LLM gateways often produces misleading results if you only look at average latency or hit rate limits, making it hard to know where the real throughput ceiling is.

Core Features & Use Cases

• Async concurrency measurement (asyncio.gather + httpx): run deterministic, CI-friendly benchmarks to compute p50/p95/p99 latency and RPS precisely.
• Locust-based distributed verification: produce dashboard-friendly reports and validate results against the asyncio method.
• Key distribution + RPS ceiling methodology: use multiple keys to avoid RPM throttling and ramp concurrency to detect the throughput “knee”.
• Capacity decision metrics: combine RPS/TPS, success rate, p99 latency, and basic GPU/memory signals to decide sustainable concurrency.

Quick Start

Run a targeted concurrent load test and generate a benchmark report by providing multiple API keys via KEY_LIST and setting a target concurrency and requests-per-user.