skills.rest
DMT123
DMT123COMMUNITY
Explore all skills by this author →
GitHub

hugging-face-evaluation

Manage model evaluation results in Hugging Face model cards.

Updated Mar 2, 2026
Search Tags:#benchmarking#evaluation#huggingface#model-index#vllm#model cards
One-click install
npx skills add https://github.com/DMT123/claude-skills-collection --skill hugging-face-evaluation-dmt123
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: hugging-face-evaluation
Source: https://github.com/DMT123/claude-skills-collection/tree/main/plugins/huggingface/hugging-face-evaluation
Command: npx skills add https://github.com/DMT123/claude-skills-collection --skill hugging-face-evaluation-dmt123

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires huggingface-hub, markdown-it-py, python-dotenv, pyyaml, requests, inspect-ai, inspect-evals, openai, lighteval[accelerate,vllm], vllm, torch, transformers, accelerate, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill streamlines the process of adding and managing evaluation results for Hugging Face models, ensuring consistency and integration with the broader Hugging Face ecosystem.

Core Features & Use Cases

  • Extract from README: Parses evaluation tables directly from model README files.
  • Import from Artificial Analysis: Fetches benchmark scores via API for seamless integration.
  • Model-Index Management: Generates and updates model-index.json for leaderboard integration.
  • Custom Evaluations: Supports running custom model evaluations using vLLM or lighteval.
  • Use Case: Automatically update your model's benchmark scores on Hugging Face Hub after a new evaluation run, or contribute scores from Artificial Analysis to a community model.

Quick Start

Use the hugging-face-evaluation skill to inspect evaluation tables in the README of the 'username/model-name' repository.

Frequently Asked Questions about hugging-face-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I add benchmark evaluation results to a Hugging Face model card?▼

To add benchmark evaluation results to a Hugging Face model card, you can parse evaluation tables directly from README files or import scores via the Artificial Analysis API. The skill formats these results into a compatible model-index metadata structure for leaderboard integration.

Can I run custom model evaluations using vLLM and lighteval for Hugging Face Hub?▼

Yes, you can run custom model evaluations using vLLM or lighteval. The skill integrates these frameworks to execute evaluations and automatically manages the output data for inclusion in your Hugging Face model-index metadata.

How does model-index metadata integration work for Hugging Face leaderboards?▼

Model-index metadata integration works by generating and updating a `model-index.json` file from your evaluation results. This process ensures your benchmark scores adhere to Papers with Code specifications, enabling seamless display on Hugging Face leaderboards.

Does this evaluation tool support importing scores from the Artificial Analysis API?▼

Yes, the tool supports importing benchmark scores directly from the Artificial Analysis API. It fetches the evaluation data via API requests and integrates those scores into your Hugging Face model card and model-index metadata.

What is the best way to extract evaluation tables from a Hugging Face README?▼

The best way to extract evaluation tables from a Hugging Face README is to parse the markdown files directly. The skill automatically reads the README structure, extracts the benchmark data, and formats it into the required model-index metadata.

Do I need vLLM and torch installed to run custom Hugging Face evaluations?▼

Yes, you need vLLM, torch, and transformers installed to run custom evaluations locally. These dependencies are required to execute the evaluation scripts and generate the benchmark results for your Hugging Face model cards.

Related Skills

Explore More Software Engineering Skills→
zeronx798zeronx798

whisper

Transcribe multilingual audio files into text using OpenAI's Whisper model.

Community
Intermediate
patronus-aipatronus-ai
2

hugging-face-evaluation-manager

Extract README evaluation tables into model-index YAML for Hugging Face model cards.

Official
Advanced
niznik-devniznik-dev
16

create-inspect-task

Create custom inspect-ai evaluation tasks with experiment-guided and standalone modes.

Community
Intermediate
POWERFULMOVESPOWERFULMOVES
1

hugging-face-evaluation-manager

Extract evaluation tables from READMEs and import benchmark scores from the Artificial Analysis API.

Community
Advanced