eval

To rank agent results in an AgentHub session, apply metric-based evaluation, LLM-based comparison, or a hybrid ranking approach to evaluate multiple agents and determine a winning output.

Metric-based evaluation runs a per-agent command to quantitatively measure performance, while LLM judge mode compares agent result diffs and qualitative verdicts. Hybrid ranking combines both approaches to determine a robust winner.

Yes, you can use LLM judge mode to compare results with diffs and qualitative verdicts. This allows an LLM to directly rank agents and determine a winner within an AgentHub session containing multiple agents.

Yes, a configured evaluation command is required to perform metric-based evaluation across agent worktrees. Without it, you can still use LLM judge mode or hybrid ranking if you prefer qualitative comparison.

Hybrid ranking determines the best agent result by combining quantitative metric scores from your evaluation command with qualitative LLM judge assessments, ensuring a robust winner is selected from the session.

Running metric-based evaluation requires an AgentHub session with multiple agents and a configured evaluation command. The scripts execute within each agent's worktree to measure and rank their performance outputs.