gen-eval

Run generator-evaluator tests against interface descriptors for live or local services.

4|1|Updated Jan 16, 2026

Search Tags:#automation #testing #cli #evaluation #scenarios #descriptor #gen-eval

Ready to run

npx skills add https://github.com/jankneumann/agentic-coding-tools --skill gen-eval

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: gen-eval
Source: https://github.com/jankneumann/agentic-coding-tools/tree/main/.agents/skills/gen-eval
Command: npx skills add https://github.com/jankneumann/agentic-coding-tools --skill gen-eval

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill automates generation, execution, and evaluation of test scenarios derived from interface descriptors to detect regressions, coverage gaps, and incorrect behavior in live or local services.

Core Features & Use Cases

Descriptor-driven Scenario Generation: Auto-detects evaluation/gen_eval descriptor YAML files and generates test scenarios from interface definitions.
Multi-mode Execution: Supports template-only, cli-augmented, and sdk-only modes with configurable time and cost budgets and parallel scenario execution.
Deterministic Reporting & Integration: Produces markdown and JSON reports, per-scenario metrics, exit-code pass/fail semantics, and integrates with validation and feature-exploration workflows to triage failures.

Quick Start

Run the gen-eval skill to auto-detect a descriptor and execute template-only generator-evaluator tests with default settings.

Frequently Asked Questions about gen-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I automate testing for microservices using interface descriptors?▼

Descriptor-driven scenario generation validates microservice behavior by auto-detecting YAML interface definitions and automatically generating test scenarios to detect regressions, coverage gaps, and incorrect behavior in live or local services.

What is a generator-evaluator test for CLI and SDK validation?▼

Generator-evaluator testing validates CLI and SDK behavior by generating test scenarios from interface descriptors and evaluating execution results. It supports template-only, cli-augmented, and sdk-only modes to verify correct service behavior.

Can I run descriptor-driven tests in parallel with time and cost budgets?▼

Yes, descriptor-driven scenario execution supports parallel test runs with configurable time and cost budgets across template-only, cli-augmented, and sdk-only modes to manage resource limits during validation.

Do I need a Python runtime to run descriptor-driven scenario generation?▼

Yes, descriptor-driven scenario generation requires a Python runtime with the evaluation module available. You also need a descriptor YAML file in the evaluation directory, plus optional CLI or SDK credentials for augmented modes.

How do I integrate generator-evaluator test results into triage workflows?▼

Generator-evaluator test results integrate into triage workflows through deterministic markdown and JSON reporting. The system produces per-scenario metrics and exit-code pass/fail semantics to streamline failure validation and feature-exploration workflows.

What are the limitations of template-only generator-evaluator testing?▼

Template-only generator-evaluator testing generates scenarios solely from interface descriptors without CLI or SDK augmentation. It cannot validate live service behavior requiring authentication credentials or runtime SDK interactions.