llm-eval-multi-model

Benchmark multiple LLM endpoints concurrently on latency, token usage, and response quality.

Updated Apr 4, 2026

Search Tags:#llm-as-judge #llm evaluation #token usage #latency metrics #model benchmarking #asyncio concurrency

Ready to run

npx skills add https://github.com/saintgo7/claude-skills --skill llm-eval-multi-model

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: llm-eval-multi-model
Source: https://github.com/saintgo7/claude-skills/tree/main/llm-eval-multi-model
Command: npx skills add https://github.com/saintgo7/claude-skills --skill llm-eval-multi-model

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires httpx, and includes scripts (resource) components.

What problem does it solve?

This Skill solves the challenge of comparing multiple LLM models fairly by measuring latency, throughput, token usage, and response quality under identical prompts and sampling settings.

Core Features & Use Cases

Parallel multi-endpoint evaluation: Sends the same prompt to multiple LLM endpoints concurrently to compare performance at the same time.
Production-style latency and token metrics: Captures TTFT, TPOT/throughput, and prompt/completion token usage with p50/p95/p99 summaries and warm-up handling.
Quality scoring options: Supports ground-truth grading for deterministic tasks and optional LLM-as-judge for subjective quality comparisons, including tool-calling accuracy evaluation patterns.

Quick Start

Run the evaluation by calling the skill install script and then request the AI to compare two models with the same prompt set, collecting latency and quality metrics into a single report.

Frequently Asked Questions about llm-eval-multi-model

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I benchmark multiple LLMs side-by-side for latency and token usage?▼

To benchmark multiple LLMs side-by-side, run the same prompts concurrently across endpoints with identical sampling settings to capture TTFT, throughput, and token usage. The evaluation aggregates p50/p95/p99 latency metrics while excluding warm-up runs to ensure fair comparisons.

What is LLM-as-judge evaluation and when should I use it for model benchmarking?▼

LLM-as-judge evaluation is an optional scoring workflow for subjective quality comparisons during model benchmarking. Use it when ground-truth grading is unavailable, such as evaluating bilingual output consistency, coding ability, or tool-calling accuracy across different vendor model families.

Does this multi-model evaluation tool support asyncio concurrency for parallel API requests?▼

Yes, asyncio concurrency is supported to send the same prompt to multiple LLM endpoints concurrently. This parallel multi-endpoint evaluation approach measures production-style latency and token metrics simultaneously, ensuring accurate side-by-side performance comparisons.

How do I perform regression validation across different LLM vendor families?▼

Perform regression validation across LLM vendor families by running repeated prompt sets with deterministic sampling controls. The benchmark isolates warm-up runs and calculates percentile-based statistics to detect latency, token usage, and quality degradations between model versions.

Do I need httpx installed to run concurrent LLM endpoint evaluations?▼

Yes, httpx is a required dependency to run concurrent LLM endpoint evaluations. It handles the asynchronous HTTP requests necessary for sending prompts to multiple models simultaneously and capturing production-style latency metrics like TPOT and TTFT.