create-eval

Name: create-eval
Availability: InStock
Author: dltHub

Scaffold trigger evaluation setups for toolkit skills with test workspaces and queries.

53|5|Updated Feb 17, 2026

Search Tags:#automation #testing #evaluation #ai assistant #skill triggers #workbench

Ready to run

npx skills add https://github.com/dlt-hub/dlthub-ai-workbench --skill create-eval

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: create-eval
Source: https://github.com/dlt-hub/dlthub-ai-workbench/tree/main/.claude/skills/create-eval
Command: npx skills add https://github.com/dlt-hub/dlthub-ai-workbench --skill create-eval

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires uv, list_skill_descriptions.py, create_eval_workspace.py, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill helps ensure that your AI assistant correctly understands when to use a specific tool (skill) by setting up automated tests for its trigger conditions.

Core Features & Use Cases

Trigger Testing: Create a dedicated environment to test if a skill's description accurately matches user requests.
Accuracy Measurement: Generate realistic "should-trigger" and "should-not-trigger" queries to measure and improve skill recall and precision.
Use Case: Before deploying a new skill for generating Python code, use this to create a suite of tests that verify it only activates for code-related requests and not for general conversation.

Quick Start

Create an evaluation for the 'rest-api-pipeline' toolkit's 'create-pipeline' skill.

Frequently Asked Questions about create-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I test AI assistant skill triggers for accuracy?▼

Testing skill triggers involves locating a skill's definition and creating an evaluation directory with test workspaces featuring varying toolkit configurations. The setup generates realistic trigger and non-trigger queries based on skill descriptions to measure activation precision and recall.

What is automated trigger evaluation for AI assistants?▼

Automated trigger evaluation is the process of verifying an AI assistant correctly understands when to use a specific tool. It measures whether a skill's description accurately matches user requests by testing it against generated queries that should and should not activate the skill.

How do I generate should-trigger and should-not-trigger queries for my skill?▼

You can generate should-trigger and should-not-trigger queries by using a skill's description to automatically create realistic test cases. This involves scaffolding an evaluation setup that builds test workspaces with varying configurations to validate the assistant's matching accuracy.

Do I need uv to set up a skill evaluation workspace?▼

Yes, uv is required as a dependency to set up the skill evaluation workspace. The toolkit relies on uv to scaffold the evaluation directory, build test workspaces with varying configurations, and prepare the environment for evaluating skill triggering accuracy.

Can I measure precision and recall for toolkit skill descriptions?▼

Yes, you can measure precision and recall for toolkit skill descriptions by scaffolding a trigger evaluation setup. The system generates test workspaces and query sets to verify your assistant only activates the correct skill for relevant user requests.

What's the best way to verify a skill only activates for specific user requests?▼

The best way to verify a skill only activates for specific user requests is to create an automated evaluation suite. By generating realistic trigger and non-trigger queries based on skill descriptions, you can test the assistant's matching accuracy before deployment.