Score dialogue bundles against an 1820 rural Ireland authenticity rubric.
npx skills add https://github.com/dmooney/Rundale --skill rundale-bench-judge
Please help me install this Agent Skill. Skill: rundale-bench-judge Source: https://github.com/dmooney/Rundale/tree/main/.agents/skills/rundale-bench-judge Command: npx skills add https://github.com/dmooney/Rundale --skill rundale-bench-judge
It solves the problem of consistently evaluating and ranking AI-generated dialogue by applying a shared rubric to each prompt/response pair and producing machine-usable scoring output.
Ask an AI agent to run the rundale-bench-judge on a bundle JSON file from rundale-bench/.bench-queue/pending and return scores as a single JSON object only.
High-intent search queries and answers about installing and using this skill.
A rubric-based LLM evaluation scores each candidate response across fixed axes like character, authenticity, language, responsiveness, and craft. It reads a bundle JSON of dialogue pairs and returns strictly validated JSON with 1-5 scores and rationales per item.
To batch score dialogue bundles, provide a bundle JSON containing multiple prompt-response pairs to a judge process. It applies a shared rubric to each item and returns one aggregated JSON object with constrained 1-5 axis scores, overall scores, and rationales.
Yes, you can detect non-Latin script in dialogue responses during LLM evaluation. The judge flags non-Latin script usage within the JSON output to support reliable downstream data ingestion and verify language authenticity constraints.
Yes, strict JSON output for dialogue judging requires reading a judge system prompt file. This file contains the evaluation contract and rubric instructions needed to score prompt-response pairs and echo the rubric hash unchanged in the result.
A limitation of automated rubric scoring for dialogue authenticity is its fixed evaluation scope, such as an 1820 rural Ireland authenticity benchmark. It requires a specific bundle JSON input and system prompt file, returning strict JSON only without alternative output formats.
m2ai-portfolioGenerate a structured quality rubric with pass/fail criteria and calibration steps.
NickCrewScore early-stage concepts by impact, delight, and effort, then rank and select a top concept for a one-day spike.
dmooneyEvaluate Rundale model quality with Sonnet 4.6 rubric scoring.
enigmaicon-engScore or compare LLM outputs with rubric-based LLM-as-judge evaluation and structured JSON results.
