issue-to-eval

Convert GitHub issues into standardized benchmark evaluation JSON.

93|23|Updated Mar 4, 2026

Search Tags:#github #ai testing #benchmarking #json generation #issue parsing

Ready to run

npx skills add https://github.com/RConsortium/pharma-skills --skill issue-to-eval

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: issue-to-eval
Source: https://github.com/RConsortium/pharma-skills/tree/main/_automation/issue-to-eval
Command: npx skills add https://github.com/RConsortium/pharma-skills --skill issue-to-eval

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires python, gh, and includes scripts (resource) components.

What problem does it solve?

This Skill automates the conversion of GitHub issues into standardized benchmark evaluation JSON, streamlining the process of AI agent iteration and testing.

Core Features & Use Cases

GitHub Issue Parsing: Extracts necessary components from GitHub issues.
Benchmark JSON Generation: Constructs JSON specifications for benchmark graders.
Use Case: When a user provides a GitHub issue with benchmark data, this Skill can automatically convert it into a JSON format that is ready for evaluation by AI agents.

Quick Start

Use the issue-to-eval skill to convert a GitHub issue into a benchmark JSON: issue-to-eval convert <issue-url>

Frequently Asked Questions about issue-to-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I convert GitHub issues into benchmark JSON for AI testing?▼

To convert GitHub issues into benchmark JSON for AI testing, use a tool that parses issue templates to extract target skills, language, and assertions, generating standardized evaluation JSON.

What components are extracted from GitHub issues for AI agent benchmarking?▼

For AI agent benchmarking, the extracted components from GitHub issues include target skills, programming language, and assertions needed to construct JSON specifications for benchmark graders.

Do I need Python and GitHub CLI to generate benchmark evaluation JSON from issues?▼

Yes, you need Python and GitHub CLI installed, as they are required dependencies for fetching GitHub issues and processing the data into standardized benchmark evaluation JSON.

How do I automate benchmark conversion for AI evaluation using GitHub issue templates?▼

Automate benchmark conversion for AI evaluation by running a script that fetches a GitHub issue via URL, parses its template fields, and outputs a JSON file ready for benchmark graders.

What is the best way to prepare GitHub issues for AI agent evaluation?▼

The best way to prepare GitHub issues for AI agent evaluation is to use standardized issue templates so target skills and assertions can be automatically parsed into benchmark JSON specifications.

Can I use GitHub CLI to fetch issue data for benchmark JSON generation?▼

Yes, you can use GitHub CLI to fetch issue data, which is then processed by Python scripts to parse the issue components and generate the standardized benchmark evaluation JSON.