Evaluate AI skill trigger accuracy against test queries using Python scripts.
npx skills add https://github.com/dlt-hub/dlthub-ai-workbench --skill run-eval
Please help me install this Agent Skill. Skill: run-eval Source: https://github.com/dlt-hub/dlthub-ai-workbench/tree/main/.claude/skills/run-eval Command: npx skills add https://github.com/dlt-hub/dlthub-ai-workbench --skill run-eval
This Skill helps improve AI assistant accuracy by evaluating how well a skill's description triggers for various user requests, identifying and fixing issues like missed triggers or incorrect activations.
Use the run-eval skill to test the 'run-eval' skill in the 'workbench' toolkit.
High-intent search queries and answers about installing and using this skill.
To evaluate AI skill trigger accuracy, run predefined test queries against a skill description to measure precision and recall, identifying consistent misses and false triggers. This analysis categorizes trigger failures and proposes description improvements for better AI understanding.
Your AI skill fails to trigger correctly due to consistent misses, competition losses, or false triggers. Evaluating the skill description against test queries identifies these specific trigger failures and provides actionable insights to refine the description for accurate activations.
To test for skill trigger clashes, run a clash analysis using predefined test queries against multiple skill descriptions. This identifies when multiple skills incorrectly trigger for the same query and proposes description improvements to prevent competition losses.
Yes, you can use Python scripts to analyze AI skill trigger performance. The evaluation requires Python scripts to execute tests within specified toolkits and workspaces, analyzing trigger performance data to identify failures and propose description improvements.
The best way to refine AI skill descriptions after updates is to run trigger evaluations using test queries. This ensures updated descriptions correctly capture new user intents without negatively impacting existing trigger performance, providing actionable insights for further refinement.
Yes, evaluating skill triggers requires a specified workspace and toolkit environment. The evaluation executes Python scripts to analyze trigger performance data within these boundaries, ensuring accurate measurement of precision and recall against predefined test queries.
dlt-hubScaffold trigger evaluation setups for toolkit skills with test workspaces and queries.
rd162Create, evaluate, and benchmark Claude skills with SKILL.md generation.
cockroachdbReduce CockroachDB roachtest failure logs to minimal SQL reproduction scripts.
thegoodpartyAnalyze entity resolution match results for coverage, false positives, and false negatives.
