Generate Harbor-compatible CodeScaleBench task files and suite configurations.
npx skills add https://github.com/sourcegraph/CodeScaleBench --skill scaffold-task
Please help me install this Agent Skill. Skill: scaffold-task Source: https://github.com/sourcegraph/CodeScaleBench/tree/main/skills/scaffold-task Command: npx skills add https://github.com/sourcegraph/CodeScaleBench --skill scaffold-task
This Skill streamlines the creation of new benchmark tasks and entire benchmark suites for the CodeScaleBench project, significantly reducing the manual effort involved in setting up new evaluation scenarios.
task.toml, instruction.md, Dockerfile, test.sh, etc.) for both SDLC and org-scale tasks.Use the scaffold-task skill to create a new SDLC task in an existing suite.
High-intent search queries and answers about installing and using this skill.
Scaffold benchmark tasks for CodeScaleBench by interactively defining task inputs to automatically generate required files like task.toml, Dockerfiles, and test scripts. This automation streamlines the creation of both SDLC and org-scale evaluation scenarios.
Creating a benchmark suite generates essential configuration files, run scripts, task.toml, instruction.md, Dockerfiles, and test.sh. This boilerplate generation reduces manual setup effort for new evaluation scenarios.
Yes, you can create org-scale tasks with this scaffolding tool. It supports generating configurations and registration entries for both SDLC and org-scale task types within the CodeScaleBench project.
The interactive workflow guides you through a series of questions to gather all required inputs for task generation. By defining task parameters and specifying involved repositories, it automates the generation of boilerplate files.
The best way to add a cross-repository dependency tracing task is to use an interactive scaffolding process to define the task, specify repositories, and generate boilerplate files, which you then customize with specific instructions and verification steps.
No, you do not need to manually write Dockerfiles for new benchmark tasks. The scaffolding process automates the generation of Dockerfiles, test scripts, and task configuration files to streamline the benchmark development workflow.
KaiserWhoLearnsAutomate agent performance evaluation with Harbor framework tasks.
yogogml0035-cpuGenerate test.sh and lint.sh scripts from AGENTS.md project configurations.
sourcegraphGenerate aggregate CodeScaleBench evaluation reports from Harbor runs in markdown, JSON, and CSV.
sourcegraphReconcile task metadata between selected_benchmark_tasks.json and task.toml files.