generate-eval

Name: generate-eval
Availability: InStock
Author: Tangle

Author and certify eval tasks measuring API detail accuracy.

3|Updated May 3, 2026

Search Tags:#code evaluation #task authoring #eval task #API measurement #library certification

Ready to run

npx skills add https://github.com/tangle-network/agent-runtime --skill generate-eval

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: generate-eval
Source: https://github.com/tangle-network/agent-runtime/tree/main/skills/generate-eval
Command: npx skills add https://github.com/tangle-network/agent-runtime --skill generate-eval

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill assists in authoring and certifying execution-certifiable, search-discriminating eval tasks for a given target, ensuring accurate API detail measurement.

Core Features & Use Cases

Task Authoring: Author eval tasks that measure whether web search aids a coding agent in obtaining correct API details.
Certification: The runtime certifies the candidate tasks against both grounding and discrimination gates.
Use Case: For a library version or release notes, create a task that an independent certifier will validate against the real target.

Quick Start

Use the generate-eval skill to create an eval task for the library '[email protected]' and output the candidate JSON object to 'task.json'.

Frequently Asked Questions about generate-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I create eval tasks to measure precise API details for a specific library version?▼

To create eval tasks for API detail measurement, you author tasks with specific schema fields, setup commands, and reference code targeting a library version or release notes, then output the candidate JSON object.

What does it mean to certify an eval task against grounding and discrimination gates?▼

Certifying an eval task against grounding and discrimination gates means an independent runtime validates that the candidate task is accurately grounded and effectively discriminates whether web search aids a coding agent against the real target.

Can I use this to generate eval tasks based on release notes and repository documentation?▼

Yes, you can generate eval tasks based on release notes and repository documentation, as the task authoring process explicitly focuses on using library versions, repositories, or release notes to ensure accurate API detail measurement.

What schema fields and setup commands do I need to author a search-discriminating eval task?▼

Authoring a search-discriminating eval task requires providing specific schema fields, setup commands, and reference code to ensure the task is execution-certifiable and can accurately measure if web search aids a coding agent.

How do I validate that my eval task is execution-certifiable before running it against a real target?▼

You validate that an eval task is execution-certifiable by using the runtime to certify the candidate tasks against both grounding and discrimination gates, ensuring it meets the required standards before testing against the real target.