benchmark-models

Cross-model benchmarking runs the same prompt through Claude, GPT, and Gemini to measure latency, token usage, cost, and output quality. It uses a local gstack-model-benchmark binary to execute comparisons and generate a JSON or table output.

Yes, cross-model benchmarking supports prompt selection from a text string, file path, or existing skill. You select the prompt source and the benchmark runs it against authed models to compare results.

Yes, you need authenticated access to model providers like Claude, GPT via Codex CLI, and Gemini. The benchmark checks provider availability and authed status before running prompts.

Output quality evaluation uses an optional external judge evaluator during the benchmark. When enabled, the judge assesses the generated outputs alongside latency, token, and cost metrics.

Yes, the benchmark includes dry-run checks as a safeguard. This allows you to verify prompt selection and provider availability before fully executing the cross-model comparison.

Limitations include dependency on provider availability and judge availability for quality assessment. Token usage and cost reporting are optional and require successful authentication across all target models.