trainer-election

Elect the strongest prompt or skill candidate from scored evaluation workspace artifacts.

Updated Mar 25, 2026
One-click install
npx skills add https://github.com/Tyler-R-Kendrick/copilot-auto-training --skill trainer-election-tyler-r-kendrick
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: trainer-election
Source: https://github.com/Tyler-R-Kendrick/copilot-auto-training/tree/main/skills/trainer-election
Command: npx skills add https://github.com/Tyler-R-Kendrick/copilot-auto-training --skill trainer-election-tyler-r-kendrick

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve? After running multiple prompt or skill candidates through an evaluation workspace, teams need a deterministic way to pick a winner from existing grading, timing, and benchmark artifacts without rerunning optimization or inventing missing scores. ## Core Features & Use Cases - Artifact-based leader selection: Reads existing grading.json and timing.json run artifacts across config directories such as with_skill, without_skill, and old_skill, falling back to benchmark.json only when raw runs are unavailable. - Coverage-aware scoring: Penalizes incomplete eval coverage so partially graded candidates cannot beat fully validated ones, with tie-breakers on raw score, errors, time, tokens, and stable name ordering. - Explainable JSON output: Returns the winner, winning prompt text or file path, per-candidate summaries, selection source, and manifest traceability fields. - Use Case: After an optimization loop produces three candidate prompts scored against an authored evals/evals.json manifest, run the election script on the latest iteration to pick the leader and persist the reasoning for later review. ## Quick Start Run the election script with your workspace directory, for example: python skills/trainer-election/scripts/run_election.py <workspace_dir> --iteration 3 --manifest-file evals/evals.json.

Frequently Asked Questions about trainer-election

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I pick the best prompt candidate from an eval workspace?

Run the run_election.py script with your workspace directory and optionally an iteration number and evals.json manifest path. It aggregates pass rates from grading.json artifacts, penalizes incomplete coverage, and returns the winning configuration as JSON.

What workspace layouts does the election runtime accept?

It accepts a workspace root with iterations/iteration-N directories, a direct iteration directory, legacy iteration-N folders, a direct eval directory, and iteration layouts that keep evals at the top level or under a runs/ subdirectory.

Can election run if only benchmark.json exists?

Yes, benchmark.json is used as a fallback when raw grading.json and timing.json run artifacts are unavailable. The result marks selection_source as benchmark so callers know which data drove the decision.

How are candidates with partial eval coverage handled?

Candidates that only covered part of the expected eval set receive a penalty equal to one minus their coverage ratio, subtracted from their raw mean pass rate. This prevents partially graded candidates from beating fully validated ones by omission.

When should I not use trainer-election?

Do not use it to gather datasets, synthesize evals, optimize prompts, or run missing evaluations from scratch. It is strictly a selection pass over candidates that have already been run and graded, and it stops with an error if no scored runs exist.