Verified:GitHub

rundale-bench

Evaluate Rundale model quality with Sonnet 4.6 rubric scoring.

Updated Mar 18, 2026

Search Tags:#A/B testing #benchmarking #performance metrics #model evaluation #cost accounting #rubric scoring #LLM judging

Ready to run

npx skills add https://github.com/dmooney/Rundale --skill rundale-bench

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: rundale-bench
Source: https://github.com/dmooney/Rundale/tree/main/.agents/skills/rundale-bench
Command: npx skills add https://github.com/dmooney/Rundale --skill rundale-bench

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

It removes the guesswork of comparing and evaluating Rundale model outputs by providing repeatable, rubric-scored quality benchmarks with cost and performance reporting.

Core Features & Use Cases

Targeted quality evaluation (bench mode): Runs all relevant slices (dialogue, intent, reaction, tier2-sim, tier3-sim, gaeilge) and a perf probe for a single model+provider target, using Sonnet 4.6 judges to score outcomes and then ingests results into a final report.
Queue-based judging (drain-queue mode): Resumes or completes evaluation by draining already-produced judging bundles from an on-disk queue without re-running generation work.
Dialogue-only blind A/B/N (eval-dialogue mode): Generates dialogue samples with Opus and uses a Sonnet 4.6 judge to score dialogue candidates across a 5-axis rubric, including per-candidate cost.
Practical failure handling: Flags provider quirks (e.g., high error rates) and supports re-dispatching missing or failed judge bundles before finalizing aggregates.

Quick Start

Run model-quality evaluation with: use the skill to benchmark a single target by saying: evaluate <model@base_url#env:VAR> on <provider>.

Frequently Asked Questions about rundale-bench

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I benchmark LLM model quality end-to-end using rubric scoring?▼

To benchmark LLM model quality end-to-end, this Skill runs targeted evaluation slices and performance probes, dispatching Sonnet 4.6 judge subagents to score outputs against a rubric and then ingesting finalized judgments with validations into a cost-aware report.

What is the best way to resume A/B testing after a judge subagent fails?▼

The best way to resume A/B testing after failures is using drain-queue mode, which scores existing on-disk judging bundles without re-running generation work. It supports re-dispatching missing or failed judge bundles before finalizing aggregates.

How do I run a single-target A/B benchmark for a specific model and provider?▼

You can run a single-target A/B benchmark by specifying the target as <model@base_url#env:VAR> on <provider>. Bench mode then executes all relevant slices, including dialogue, intent, reaction, and simulation probes, before ingesting scored results.

Can I evaluate dialogue-only outputs without running full performance probes?▼

Yes, you can evaluate dialogue-only outputs using eval-dialogue mode, which performs blind A/B/N testing focused exclusively on dialogue candidates. It scores across a 5-axis rubric and calculates per-candidate costs without executing the full performance probe suite.

What happens if a provider has high error rates during model evaluation?▼

If a provider has high error rates during model evaluation, the Skill flags these provider quirks and supports re-dispatching missing or failed judge bundles. This error-aware reporting ensures you can finalize aggregates without losing completed judging work.