skills.rest
mashharuki
mashharukiCOMMUNITY
Explore all skills by this author →
Website𝕏@haruki_web3GitHub

hugging-face-evaluation

Extract evaluation tables from Hugging Face model README files.

Updated Feb 28, 2026
Search Tags:#benchmarking#evaluation#huggingface#vllm#leaderboard#lighteval#model card
One-click install
npx skills add https://github.com/mashharuki/Mistral-Worldwide-Hackathon --skill hugging-face-evaluation-mashharuki
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: hugging-face-evaluation
Source: https://github.com/mashharuki/Mistral-Worldwide-Hackathon/tree/main/.agents/skills/hugging-face-evaluation
Command: npx skills add https://github.com/mashharuki/Mistral-Worldwide-Hackathon --skill hugging-face-evaluation-mashharuki

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires huggingface-hub, markdown-it-py, python-dotenv, pyyaml, requests, inspect-ai, inspect-evals, openai, vllm, torch, transformers, accelerate, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill streamlines the process of adding and managing evaluation results for Hugging Face models, making it easier to track performance and contribute to leaderboards.

Core Features & Use Cases

  • Extract from README: Automatically parse evaluation tables directly from your model's README file.
  • Import from Artificial Analysis: Fetch benchmark scores directly from the Artificial Analysis API.
  • Run Custom Evaluations: Execute evaluations using vLLM or lighteval on your own hardware or Hugging Face Jobs.
  • Use Case: After training a new language model, use this Skill to automatically extract its benchmark scores from the README, add them to the model card, and even create a pull request for review.

Quick Start

Use the hugging-face-evaluation skill to inspect evaluation tables in the 'username/model-name' repository.

Frequently Asked Questions about hugging-face-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract benchmarking evaluation results from a Hugging Face model README?▼

You can extract evaluation results by parsing the model-index metadata format directly from the README file. This Skill uses markdown-it-py to automatically pull benchmark scores from the model card, enabling easy tracking and leaderboard integration.

Can I run custom evaluations using vLLM and lighteval for Hugging Face models?▼

Yes, you can run custom evaluations using vLLM or lighteval. The Skill supports executing benchmarks on your own hardware or scaling execution through Hugging Face Jobs to evaluate model performance.

How do I import benchmark scores from Artificial Analysis into a Hugging Face model card?▼

You can import benchmark scores by fetching data directly from the Artificial Analysis API. The Skill manages these evaluation results and integrates them into the Hugging Face model card using the model-index metadata format.

Does this Skill support creating pull requests for Hugging Face model card updates?▼

Yes, after extracting evaluation tables or adding benchmark scores, the Skill can create a pull request for review. This streamlines updating the Hugging Face model card within the repository.

What dependencies are required to parse Hugging Face evaluation tables and interact with the API?▼

You need huggingface-hub, markdown-it-py, python-dotenv, pyyaml, and requests for parsing and API interaction. Additional dependencies like vllm, torch, transformers, and accelerate are required for running custom evaluations.

How does leaderboard integration work with Hugging Face evaluation results?▼

Leaderboard integration works by utilizing the model-index metadata format in the README. The Skill extracts and structures benchmark scores so they can be automatically recognized and displayed on Hugging Face leaderboards.

Related Skills

Explore More Software Engineering Skills→
niznik-devniznik-dev
16

create-inspect-task

Create custom inspect-ai evaluation tasks with experiment-guided and standalone modes.

Community
Intermediate
POWERFULMOVESPOWERFULMOVES
1

hugging-face-evaluation-manager

Extract evaluation tables from READMEs and import benchmark scores from the Artificial Analysis API.

Community
Advanced
tripplen23tripplen23
1

huggingface-community-evals

Evaluate Hugging Face Hub models locally with inspect-ai and lighteval.

Community
Advanced
FacuMFacuM

hugging-face-evaluation

Extract evaluation tables from Hugging Face model README files.

Community
Advanced