create-eval

Testing skill triggers involves locating a skill's definition and creating an evaluation directory with test workspaces featuring varying toolkit configurations. The setup generates realistic trigger and non-trigger queries based on skill descriptions to measure activation precision and recall.

Automated trigger evaluation is the process of verifying an AI assistant correctly understands when to use a specific tool. It measures whether a skill's description accurately matches user requests by testing it against generated queries that should and should not activate the skill.

You can generate should-trigger and should-not-trigger queries by using a skill's description to automatically create realistic test cases. This involves scaffolding an evaluation setup that builds test workspaces with varying configurations to validate the assistant's matching accuracy.

Yes, uv is required as a dependency to set up the skill evaluation workspace. The toolkit relies on uv to scaffold the evaluation directory, build test workspaces with varying configurations, and prepare the environment for evaluating skill triggering accuracy.

Yes, you can measure precision and recall for toolkit skill descriptions by scaffolding a trigger evaluation setup. The system generates test workspaces and query sets to verify your assistant only activates the correct skill for relevant user requests.

The best way to verify a skill only activates for specific user requests is to create an automated evaluation suite. By generating realistic trigger and non-trigger queries based on skill descriptions, you can test the assistant's matching accuracy before deployment.