Convert GitHub issues into standardized benchmark evaluation JSON.
npx skills add https://github.com/RConsortium/pharma-skills --skill issue-to-eval
Please help me install this Agent Skill. Skill: issue-to-eval Source: https://github.com/RConsortium/pharma-skills/tree/main/_automation/issue-to-eval Command: npx skills add https://github.com/RConsortium/pharma-skills --skill issue-to-eval
This Skill automates the conversion of GitHub issues into standardized benchmark evaluation JSON, streamlining the process of AI agent iteration and testing.
Use the issue-to-eval skill to convert a GitHub issue into a benchmark JSON: issue-to-eval convert <issue-url>
High-intent search queries and answers about installing and using this skill.
To convert GitHub issues into benchmark JSON for AI testing, use a tool that parses issue templates to extract target skills, language, and assertions, generating standardized evaluation JSON.
For AI agent benchmarking, the extracted components from GitHub issues include target skills, programming language, and assertions needed to construct JSON specifications for benchmark graders.
Yes, you need Python and GitHub CLI installed, as they are required dependencies for fetching GitHub issues and processing the data into standardized benchmark evaluation JSON.
Automate benchmark conversion for AI evaluation by running a script that fetches a GitHub issue via URL, parses its template fields, and outputs a JSON file ready for benchmark graders.
The best way to prepare GitHub issues for AI agent evaluation is to use standardized issue templates so target skills and assertions can be automatically parsed into benchmark JSON specifications.
Yes, you can use GitHub CLI to fetch issue data, which is then processed by Python scripts to parse the issue components and generate the standardized benchmark evaluation JSON.
ybbms777Analyze GitHub issue trackers to extract recurring pain patterns and severity trends.
WUIX69Define evaluation frameworks for Claude Code tasks with pass@k metrics.
14790897Manage GitHub issues, pull requests, and CI/CD workflows via gh CLI.
surus-latGenerate synthetic benchmark datasets as JSONL input-output pairs from task specifications.
