skills.rest
Himanshu040604
Himanshu040604COMMUNITY
Explore all skills by this author →
WebsiteGitHub

hugging-face-evaluation

Extract evaluation tables from Hugging Face model README files.

Updated Mar 5, 2026
Search Tags:#evaluation#benchmark#huggingface#vllm#leaderboard#lighteval#model card
One-click install
npx skills add https://github.com/Himanshu040604/codex-skills-setup --skill hugging-face-evaluation-himanshu040604
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: hugging-face-evaluation
Source: https://github.com/Himanshu040604/codex-skills-setup/tree/main/assets/codex/skills/claude-import/skills/plugins/huggingface-skills%40claude-plugins-official/skills/hugging-face-evaluation
Command: npx skills add https://github.com/Himanshu040604/codex-skills-setup --skill hugging-face-evaluation-himanshu040604

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires huggingface-hub, markdown-it-py, python-dotenv, pyyaml, requests, inspect-ai, inspect-evals, openai, lighteval[accelerate,vllm], vllm, torch, transformers, accelerate, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill streamlines the process of adding and managing evaluation results for Hugging Face models, ensuring accurate and up-to-date performance metrics are reflected in model cards.

Core Features & Use Cases

  • Extract from README: Parses evaluation tables directly from model README files.
  • Import from Artificial Analysis: Fetches benchmark scores via API for seamless integration.
  • Run Custom Evaluations: Executes model evaluations using vLLM or lighteval on local GPUs or Hugging Face Jobs.
  • Model-Index Management: Updates model-index metadata for leaderboards and compatibility.
  • Use Case: A researcher wants to add the latest benchmark scores for their new LLM to its Hugging Face model card. They can use this Skill to automatically extract scores from their README or import them from Artificial Analysis, creating a pull request for review.

Quick Start

Use the hugging-face-evaluation skill to extract evaluation tables from the model's README file.

Frequently Asked Questions about hugging-face-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I add benchmark results to a Hugging Face model card?▼

To add benchmark results to a Hugging Face model card, you can extract evaluation tables directly from the README file or import scores via the Artificial Analysis API, then automatically generate a pull request to update the metadata.

Can I run custom LLM evaluations locally using vLLM and lighteval?▼

Yes, you can run custom LLM evaluations using vLLM or lighteval on local GPUs or Hugging Face Jobs, allowing you to compute performance metrics and seamlessly update your model-index metadata for leaderboards.

How does importing benchmark scores from Artificial Analysis work for Hugging Face models?▼

Importing benchmark scores from Artificial Analysis fetches evaluation data via its API and integrates those metrics directly into your Hugging Face model card, ensuring performance scores are accurately reflected without manual data entry.

Do I need to manually update the model-index metadata for Hugging Face leaderboards?▼

No, you do not need to manually update the model-index metadata. This process manages model-index updates automatically and integrates with the Hugging Face Hub to create pull requests for leaderboard compatibility.

What is the best way to parse evaluation tables from a Hugging Face README?▼

The best way to parse evaluation tables from a Hugging Face README is to use automated extraction tools that read the markdown files directly, pulling the performance metrics to update your model card metadata.

Can automated pull requests be created on the Hugging Face Hub for evaluation updates?▼

Yes, automated pull requests can be created on the Hugging Face Hub. The system integrates directly with the Hub to manage and submit pull requests containing your updated model evaluation results and model-index metadata.

Related Skills

Explore More Software Engineering Skills→
Dianshu-LiaoDianshu-Liao
2

hugging-face-evaluation-manager

Extract README evaluation tables into Hugging Face model-index YAML.

Community
Advanced
lidge-junlidge-jun
5

hugging-face-evaluation

Manage Hugging Face model evaluation results from READMEs and APIs.

Community
Advanced
BermudaLocalsBermudaLocals
1

serving-llms-vllm

Deploy vLLM inference servers with PagedAttention, continuous batching, and quantization.

Community
Advanced
niznik-devniznik-dev
16

create-inspect-task

Create custom inspect-ai evaluation tasks with experiment-guided and standalone modes.

Community
Intermediate