llm-eval-multi-model

To benchmark multiple LLMs side-by-side, run the same prompts concurrently across endpoints with identical sampling settings to capture TTFT, throughput, and token usage. The evaluation aggregates p50/p95/p99 latency metrics while excluding warm-up runs to ensure fair comparisons.

LLM-as-judge evaluation is an optional scoring workflow for subjective quality comparisons during model benchmarking. Use it when ground-truth grading is unavailable, such as evaluating bilingual output consistency, coding ability, or tool-calling accuracy across different vendor model families.

Yes, asyncio concurrency is supported to send the same prompt to multiple LLM endpoints concurrently. This parallel multi-endpoint evaluation approach measures production-style latency and token metrics simultaneously, ensuring accurate side-by-side performance comparisons.

Perform regression validation across LLM vendor families by running repeated prompt sets with deterministic sampling controls. The benchmark isolates warm-up runs and calculates percentile-based statistics to detect latency, token usage, and quality degradations between model versions.

Yes, httpx is a required dependency to run concurrent LLM endpoint evaluations. It handles the asynchronous HTTP requests necessary for sending prompts to multiple models simultaneously and capturing production-style latency metrics like TPOT and TTFT.