Benchmarks local LLM models for speed and output quality.
npx skills add https://github.com/MetriLLM/metrillm --skill benchmark-metrillm
Please help me install this Agent Skill. Skill: benchmark Source: https://github.com/MetriLLM/metrillm/tree/main/plugins/claude-code/skills/benchmark Command: npx skills add https://github.com/MetriLLM/metrillm --skill benchmark-metrillm
This Skill enables users to benchmark local language models, measuring their speed, responsiveness, and quality for informed decision-making.
Describe your target model to the AI, and ask to run the benchmark to see its performance metrics and quality evaluation.
High-intent search queries and answers about installing and using this skill.
To benchmark local LLM models, this Skill performs comprehensive performance measurement, quantifying tokens per second, time to first token, and memory usage. It requires bash scripting and model management tools to parse outputs and generate objective speed and responsiveness metrics.
You can evaluate local LLM output quality for coding and math tasks using this Skill. It performs comprehensive quality assessment, specifically measuring reasoning, math, coding, instruction following, and multilingual capabilities to aid developers in model evaluation.
The best way to compare multiple local LLM models for deployment is using this Skill to run objective benchmarks. It assesses both speed and output quality across different hardware and model configurations, providing the comparative metrics needed to guide your model selection.
Yes, you need bash scripting capabilities to run LLM performance comparisons across different hardware. The Skill relies on bash scripting, output parsing, and model management tools to execute benchmarks and assess both speed and output quality across various configurations.
Local model benchmarking works by executing scripts that run the target model and parsing the output to measure time to first token. This Skill performs comprehensive benchmarking to quantify this responsiveness metric alongside overall tokens per second and memory usage.
Evaluating local LLM performance and quality is limited by the specific hardware and model configurations used during the benchmark. Since it requires local execution and bash scripting for output parsing, results are constrained by your local environment and may not reflect cloud deployment performance.
davpatel605-beepEvaluate LLMs against standardized benchmarks like MMLU and GSM8K.
atholaEnforce a fixed template with sections, findings, and citations for review deliverables.
sourcegraphGenerate aggregate CodeScaleBench evaluation reports from Harbor runs in markdown, JSON, and CSV.
cxnaiveBenchmark large language models on academic and reasoning tasks with standardized prompts and metrics.
