skills.rest
stopitdan
stopitdanCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

run-evals

Run the recommendation engine evaluation suite and summarize pass rates, NDCG, and failures.

Updated Mar 25, 2026
Search Tags:#automation#evaluation#regression-testing#llm-judge#test-suite#ndcg#recommendation-engine
One-click install
npx skills add https://github.com/stopitdan/recommendagame --skill run-evals-stopitdan
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: run-evals
Source: https://github.com/stopitdan/recommendagame/tree/main/.claude/skills/run-evals
Command: npx skills add https://github.com/stopitdan/recommendagame --skill run-evals-stopitdan

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill automates running the boredgame.lol recommendation engine evaluation suite to measure recommendation quality, detect regressions, and produce a clear, prioritized summary of failures and remediation steps.

Core Features & Use Cases

  • Full and Quick Runs: Supports full-suite evaluation and quick 50-case smoke tests, with optional category filtering and concurrency controls.
  • Automated Scoring & Comparison: Uses an LLM judge to score results, computes metrics like pass rate and NDCG@10, and compares each run to the previous baseline to surface regressions.
  • Failure Analysis & Recommendations: Produces category breakdowns, top failing cases, most-missing games, constraint violation details, and concrete engineering recommendations for fixes.
  • Use Case: Ideal for engineers or QA teams validating recommendation quality after model or code changes and for tracking regressions over time.

Quick Start

Run a quick eval with the --quick flag and return a summarized report comparing results to the previous run.

Frequently Asked Questions about run-evals

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run recommendation engine evaluation suites and detect regressions automatically?▼

Run recommendation engine evaluation suites by executing the eval runner with concurrency and optional flags, which computes pass rates and NDCG to detect regressions against a previous baseline.

What metrics are included in an LLM judge evaluation report for recommendation systems?▼

An LLM judge evaluation report includes pass rates, LLM judge scores, NDCG@10, category breakdowns, top failing cases, most-missing games, and constraint violation details.

How do I execute a quick smoke test for recommendation engine quality after a deployment?▼

Execute a quick 50-case smoke test for recommendation engine quality by running the eval runner with the --quick flag, generating a summarized report comparing results to the previous run.

Can I filter recommendation engine regression testing by specific game categories?▼

Yes, you can filter recommendation engine regression testing by specific game categories using optional category filtering flags, allowing targeted evaluation subsets.

Do I need to configure environment variables before running recommendation engine test suites?▼

Yes, you must source the .env.local file before running recommendation engine test suites to ensure the eval runner executes correctly within the testing environment.

Related Skills

Explore More Software Engineering Skills→
0xHoneyJar0xHoneyJar
1

loa-credentials

Manage API credentials across environment variables, encrypted stores, and dotenv files.

Official
Intermediate
dogkeeper886dogkeeper886

analyzing-logs

Parse Robot Framework XML and HTML logs to classify failures by type.

Community
Intermediate
applied-artificial-intelligenceapplied-artificial-intelligence
81

llm-evaluation

Evaluate LLM applications with automated metrics and LLM-as-judge scoring.

Official
Advanced
seo-jinseokseo-jinseok

regulation-quality

Simulate user personas and score Regulation Q&A responses with LLM-as-Judge.

Community
Advanced