benchmark-models

To compare AI model performance, you can run the same prompt through Claude, GPT, and Gemini models to evaluate latency, tokens, cost, and optionally quality. This cross-model benchmarking provides data-driven insights for AI model selection and decision-making.

Evaluating LLM performance involves measuring latency, token usage, and cost across different models. Optionally, you can assess output quality by utilizing an LLM judge, providing comprehensive data-driven insights for the model selection process.

You do not strictly need an LLM judge to benchmark models, as performance evaluation primarily measures latency, tokens, and cost. However, an LLM judge is required as a dependency if you want to optionally assess and compare output quality.

To run a cross-model benchmark, use the /benchmark-models command with your specific prompt. This executes the prompt across Claude, GPT via Codex CLI, and Gemini, comparing their performance in terms of speed, cost, and token usage.

Comparing Claude, GPT, and Gemini models requires access to all three respective models. Specifically, GPT access is routed via the Codex CLI, and if you include quality assessments, you must also have access to an LLM judge.