benchmark-model

Benchmark AI model inference latency, throughput, and resource usage via CLI.

1.7k|68|Updated Jun 23, 2025

Search Tags:#performance #cli #latency #benchmark #throughput #model #inference

Ready to run

npx skills add https://github.com/trymirai/uzu --skill benchmark-model

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: benchmark-model
Source: https://github.com/trymirai/uzu/tree/main/agents/skills/benchmark-model
Command: npx skills add https://github.com/trymirai/uzu --skill benchmark-model

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Benchmark AI model inference performance across configurations.

Core Features & Use Cases

End-to-end benchmarking of model inference across engines, versions, and models.
Outputs metrics such as memory_used, time_to_first_token, prompt_tokens_per_second, and generate_tokens_per_second for comparison.
Use case: compare performance between models and configurations to guide optimization.

Quick Start

Run the benchmark for a chosen model folder and save the results to a file.

Frequently Asked Questions about benchmark-model

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I benchmark AI model inference performance across different configurations?▼

Benchmarking AI model inference performance involves running a CLI workflow against models located in specific directories to measure latency, throughput, and resource usage. It outputs metrics like memory_used and tokens per second to compare configurations.

What metrics are measured when evaluating model inference latency and throughput?▼

Evaluating model inference latency and throughput produces metrics including memory_used, time_to_first_token, prompt_tokens_per_second, and generate_tokens_per_second. These measurements help compare performance between models and configurations to guide optimization.

How do I run a benchmark task for a local model using the CLI?▼

To run a benchmark task using the CLI, execute the command `cargo run --release -p cli -- bench` pointing to your model folder under `./models/{ENGINE_VERSION}/{MODEL_NAME}`, provide a `benchmark_task.json` file, and specify a result path to save the output metrics.

Do I need a specific directory structure to evaluate model performance?▼

Yes, evaluating model performance requires a specific directory structure where models are located under `./models/{ENGINE_VERSION}/{MODEL_NAME}`. The benchmark task configuration must also be placed as `benchmark_task.json` within this model folder structure.

Can I compare performance between different models and engine versions?▼

Yes, you can compare performance between different models and engine versions by running the benchmark across various configurations. The resulting metrics files allow direct comparison of resource usage, latency, and token generation speeds to guide optimization decisions.

What is the best way to measure time to first token for AI models?▼

The best way to measure time to first token for AI models is running a benchmark CLI workflow that specifically captures this metric alongside prompt_tokens_per_second and generate_tokens_per_second. Results are saved to a specified file path for analysis.