skills.rest
open-thoughts
open-thoughtsOFFICIAL
Explore all skills by this author →
WebsiteGitHub

eval-standard-cleanup

Consolidate finished standard eval jobs into a SCORES.md tracker.

276|37|Updated Dec 5, 2025
Search Tags:#evaluation#consolidation#rsync#scores#lm_eval#evalchemy#seed42
One-click install
npx skills add https://github.com/open-thoughts/OpenThoughts-Agent --skill eval-standard-cleanup
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-standard-cleanup
Source: https://github.com/open-thoughts/OpenThoughts-Agent/tree/main/.claude/skills/eval-standard-cleanup
Command: npx skills add https://github.com/open-thoughts/OpenThoughts-Agent --skill eval-standard-cleanup

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Consolidates finished standard eval jobs (lm_eval/evalchemy math-suite) into the SCORES.md tracker, ensuring non-empty seed42 results are captured and scalars derived for reporting.

Core Features & Use Cases

  • Idempotent consolidation of completed eval jobs from per-model directories into SCORES.md, with safeguards to avoid duplicating entries.
  • Rsync and organize per-task results into the per-model archive and generate durable scalar partials for the table.
  • Parse and compute scalar metrics (MATH-500, AIME24, gsm8k, Raw) following the defined convention and update the tracker without DB uploads.

Quick Start

Run the eval-standard-launch workflow to consolidate completed standard eval results into the SCORES.md tracker.

Frequently Asked Questions about eval-standard-cleanup

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I consolidate lm_eval benchmark results into a single tracker?▼

You can consolidate standard eval results by running a workflow that rsyncs per-model directories, parses scalar metrics for benchmarks like MATH-500 and gsm8k, and updates the SCORES.md tracker idempotently without database registration.

What is the best way to aggregate evalchemy scores across multiple model runs?▼

Aggregating evalchemy scores involves parsing non-empty seed42 results and computing scalar metrics following defined conventions to populate a durable SCORES.md tracker table across multiple model runs.

Does the eval-standard-cleanup workflow avoid duplicating entries when re-running?▼

Yes, the consolidation workflow applies idempotent rsync safeguards to ensure completed eval jobs from per-model directories are organized without duplicating existing entries in the SCORES.md tracker.

How are scalar metrics parsed from standard math-suite benchmarks?▼

Scalar metrics for benchmarks like MATH-500, AIME24, and gsm8k are parsed by extracting non-empty seed42 results and deriving durable scalar partials for the reporting table.

Can I use this consolidation process if my eval results require database registration?▼

No, this process is designed specifically for HF-upload-only artifacts and generates scalar partials for the SCORES.md tracker without performing any database registration.

Related Skills

Explore More Data & Analytics Skills→
markwang2658markwang2658
31

evaluating-llms-harness

Evaluate LLMs across 60+ benchmarks using the lm_eval CLI.

Community
Advanced
open-thoughtsopen-thoughts
276

sft-cleanup-hf-only

Publish HF-only SFT cells to Hugging Face without database registration.

Official
Advanced
krzemienskikrzemienski
2

skill-creator

Orchestrate Claude Code skill development from SKILL.md drafting to trigger evaluation.

Community
Advanced
math-incmath-inc
1.2k

evaluating-llms-harness

Benchmark LLMs across 60+ academic tasks using lm-eval harness.

Official
Advanced