Execute standardized test cases for AI skills and aggregate grading benchmarks.
npx skills add https://github.com/JetBrains/databao-cli --skill eval-skills-jetbrains
Please help me install this Agent Skill. Skill: eval-skills Source: https://github.com/JetBrains/databao-cli/tree/main/.claude/skills/eval-skills Command: npx skills add https://github.com/JetBrains/databao-cli --skill eval-skills-jetbrains
This skill provides a structured approach to quantify and improve the quality of AI skills by running reproducible evaluations, capturing results, and guiding iterations.
Run an initial evaluation cycle to select skills, generate an iteration workspace, execute tests, and review the resulting grading and benchmarks.
High-intent search queries and answers about installing and using this skill.
Benchmarking AI skills involves executing standardized test cases twice—once with the skill loaded and once without—to compare outcomes. This process aggregates results into a benchmark, computing pass rates and delta improvements to guide iterative development.
Automating skill evaluation requires organizing evals.json files into iteration workspaces, executing reproducible cases, and saving grading outputs. This aggregates assertion outcomes and evidence into grading.json to compute pass rates.
Iterating on SKILL.md content uses aggregated benchmark results across multiple evaluation cycles. By analyzing delta improvements and feedback outputs, you can target specific deficiencies and close the loop on quality.
Yes, you can determine skills to evaluate by enumerating evals.json files or by selecting explicit targets. This generates an iteration workspace in .claude/evals-workspace/iteration-<N> to separate outputs and maintain clean contexts.
Grading AI skills works by collecting assertion outcomes and supporting evidence into a grading.json file. The system then computes pass rates from these aggregated outcomes to quantify skill quality.
Yes, organizing eval assets is required before evaluating. The process depends on structured evals.json files to generate iteration workspaces, execute cases cleanly, and save grading, benchmark, and feedback outputs for audit.
nibojGenerates evaluation cases and an evals.json for a target skill.
akbarfarooq2006Automates creation, testing, and refinement of Claude Code skills with built-in evals and benchmarking.
KimDoubleBDesign k6 load profiles with executors, VU patterns, and thresholds.
ChristineThamBenchmark AI agents across multi-system environments to evaluate integration reliability.
