Benchmark AI model inference latency, throughput, and resource usage via CLI.
npx skills add https://github.com/trymirai/uzu --skill benchmark-model
Please help me install this Agent Skill. Skill: benchmark-model Source: https://github.com/trymirai/uzu/tree/main/agents/skills/benchmark-model Command: npx skills add https://github.com/trymirai/uzu --skill benchmark-model
Benchmark AI model inference performance across configurations.
Run the benchmark for a chosen model folder and save the results to a file.
High-intent search queries and answers about installing and using this skill.
Benchmarking AI model inference performance involves running a CLI workflow against models located in specific directories to measure latency, throughput, and resource usage. It outputs metrics like memory_used and tokens per second to compare configurations.
Evaluating model inference latency and throughput produces metrics including memory_used, time_to_first_token, prompt_tokens_per_second, and generate_tokens_per_second. These measurements help compare performance between models and configurations to guide optimization.
To run a benchmark task using the CLI, execute the command `cargo run --release -p cli -- bench` pointing to your model folder under `./models/{ENGINE_VERSION}/{MODEL_NAME}`, provide a `benchmark_task.json` file, and specify a result path to save the output metrics.
Yes, evaluating model performance requires a specific directory structure where models are located under `./models/{ENGINE_VERSION}/{MODEL_NAME}`. The benchmark task configuration must also be placed as `benchmark_task.json` within this model folder structure.
Yes, you can compare performance between different models and engine versions by running the benchmark across various configurations. The resulting metrics files allow direct comparison of resource usage, latency, and token generation speeds to guide optimization decisions.
The best way to measure time to first token for AI models is running a benchmark CLI workflow that specifically captures this metric alongside prompt_tokens_per_second and generate_tokens_per_second. Results are saved to a specified file path for analysis.
trymiraiDownload test models by REPO_ID into ./models/{ENGINE_VERSION}/{MODEL_NAME}.
zli5460Deploys TensorRT configurations to optimize LLM inference on NVIDIA GPUs.
plurigridProvide cargo subcommand guidance for Rust project management workflows.
chiptoe-svgExpose locally hosted Ollama models as callable tools for NanoClaw container agents.
