evaluating-code-models
OfficialBenchmark code generation models.
AuthorOrchestra-Research
Version1.0.0
Installs0
System Documentation
What problem does it solve?
This Skill automates the evaluation of code generation models across a wide array of benchmarks, providing standardized metrics for performance comparison.
Core Features & Use Cases
- Multi-Benchmark Evaluation: Supports HumanEval, MBPP, MultiPL-E, and over 15 other benchmarks.
- Pass@k Metrics: Measures functional correctness with industry-standard pass@k metrics.
- Multi-Language Support: Evaluates models across 18 different programming languages.
- Use Case: When comparing two new code generation models, use this Skill to run them both against HumanEval and MBPP to objectively determine which performs better.
Quick Start
Evaluate the 'bigcode/starcoder2-7b' model on the HumanEval benchmark.
Dependency Matrix
Required Modules
bigcode-evaluation-harnesstransformers>=4.25.1accelerate>=0.13.2datasets>=2.6.1
Components
scriptsreferences
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: evaluating-code-models Download link: https://github.com/Orchestra-Research/AI-Research-SKILLs/archive/main.zip#evaluating-code-models Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 471,000+ vetted skills library on demand.