skills.rest
lidge-jun
lidge-junCOMMUNITY
Explore all skills by this author →
𝕏@claudeebumGitHub

hugging-face-evaluation

Manage Hugging Face model evaluation results from READMEs and APIs.

5|Updated Feb 26, 2026
Search Tags:#evaluation#benchmark#huggingface#vllm#leaderboard#lighteval#model card
One-click install
npx skills add https://github.com/lidge-jun/cli-jaw-skills --skill hugging-face-evaluation-lidge-jun
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: hugging-face-evaluation
Source: https://github.com/lidge-jun/cli-jaw-skills/tree/main/hugging-face-evaluation
Command: npx skills add https://github.com/lidge-jun/cli-jaw-skills --skill hugging-face-evaluation-lidge-jun

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires huggingface-hub, markdown-it-py, python-dotenv, pyyaml, requests, inspect-ai, inspect-evals, openai, lighteval[accelerate,vllm], vllm, torch, transformers, accelerate, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill streamlines the process of adding, managing, and running evaluations for Hugging Face models, ensuring consistent and accurate reporting of model performance.

Core Features & Use Cases

  • Extract Evaluations: Automatically parse evaluation tables from model README files.
  • Import Scores: Fetch benchmark scores directly from the Artificial Analysis API.
  • Run Custom Evaluations: Execute model evaluations using vLLM or lighteval for local GPU testing.
  • Model Card Integration: Updates model-index metadata for leaderboard integration and Papers with Code compatibility.
  • Use Case: A researcher wants to add the latest benchmark scores for their new model to its Hugging Face model card. They can use this Skill to extract scores from their README or import them from Artificial Analysis, automatically updating the model-index and creating a pull request for review.

Quick Start

Use the hugging-face-evaluation skill to inspect evaluation tables in the README of the 'username/model-name' repository.

Frequently Asked Questions about hugging-face-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I add benchmark scores to a Hugging Face model card?▼

To add benchmark scores to a Hugging Face model card, you can extract evaluation tables from the README or import scores from the Artificial Analysis API, which updates the model-index metadata for leaderboard and Papers with Code compatibility.

How do I run custom evaluations for Hugging Face models locally?▼

Run custom evaluations for Hugging Face models locally by executing tests with vLLM or lighteval on a local GPU, generating performance metrics that can be integrated directly into your model card metadata.

Can I import benchmark scores from Artificial Analysis into Hugging Face?▼

Yes, you can import benchmark scores from Artificial Analysis into Hugging Face by fetching data directly from the API, automatically updating the model-index and creating a pull request for review.

Does this evaluation tool support vLLM and lighteval for local GPU testing?▼

Yes, this tool supports vLLM and lighteval for local GPU testing, allowing you to execute custom model evaluations and generate benchmark results for your Hugging Face repositories.

What is the best way to update model-index metadata for Hugging Face leaderboards?▼

The best way to update model-index metadata for Hugging Face leaderboards is to parse evaluation tables from README files or import scores, ensuring consistent reporting and enabling automated pull request creation and validation.

How does extracting evaluation tables from a README work for Hugging Face models?▼

Extracting evaluation tables from a README works by automatically parsing the markdown content to capture benchmark data, which is then structured into the model-index format for leaderboard integration and Papers with Code compatibility.

Related Skills

Explore More Software Engineering Skills→
POWERFULMOVESPOWERFULMOVES
1

hugging-face-evaluation-manager

Extract evaluation tables from READMEs and import benchmark scores from the Artificial Analysis API.

Community
Advanced
Himanshu040604Himanshu040604

hugging-face-evaluation

Extract evaluation tables from Hugging Face model README files.

Community
Advanced
itallstartedwithaideaitallstartedwithaidea
31

ml-model-integration

Select and deploy HuggingFace Hub models with task-first filtering and evaluation.

Community
Advanced
niznik-devniznik-dev
16

create-inspect-task

Create custom inspect-ai evaluation tasks with experiment-guided and standalone modes.

Community
Intermediate