skills.rest
aisa-group
aisa-groupOFFICIAL
Explore all skills by this author →
WebsiteGitHub

hugging-face-evaluation-manager

Manage structured evaluation results in Hugging Face model cards.

90|4|Updated Feb 22, 2026
Search Tags:#benchmarking#evaluation#huggingface#vllm#lighteval#artificial-analysis#model-cards
One-click install
npx skills add https://github.com/aisa-group/skill-inject --skill hugging-face-evaluation-manager-aisa-group
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: hugging-face-evaluation-manager
Source: https://github.com/aisa-group/skill-inject/tree/main/data/skills/hf_model_evaluation
Command: npx skills add https://github.com/aisa-group/skill-inject --skill hugging-face-evaluation-manager-aisa-group

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires huggingface-hub, markdown-it-py, python-dotenv, pyyaml, requests, inspect-ai, inspect-evals, openai, vllm, torch, transformers, accelerate, lighteval, and includes scripts (resource) components.

What problem does it solve?

Manage the tedious and error-prone process of adding, formatting, and publishing structured evaluation results to Hugging Face model cards so maintainers can keep leaderboards and model-index metadata accurate and discoverable.

Core Features & Use Cases

  • README table extraction: Detect and parse evaluation tables from model READMEs and convert them into model-index YAML entries.
  • Artificial Analysis import: Fetch benchmark scores from the Artificial Analysis API and merge them into model cards with source attribution.
  • Run and submit evaluations: Launch inspect-ai or lighteval/vLLM evaluations locally or via Hugging Face Jobs and capture results into the model-index format.
  • Safe publishing workflows: Validate YAML, merge with existing model-index entries, check for open PRs before creating new ones, and optionally create pull requests to avoid spamming repositories.
  • Use Case: A model maintainer or researcher can extract README tables, import third-party benchmark scores, run additional vLLM evaluations, and produce a single validated model-index entry ready to push or open as a PR.

Quick Start

Extract evaluation tables from a model README and generate a validated model-index YAML ready to push or create a pull request on Hugging Face.

Frequently Asked Questions about hugging-face-evaluation-manager

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I add benchmark results to a Hugging Face model card?▼

Yes, you can import Artificial Analysis benchmark scores into your model card by fetching data via the Artificial Analysis API. The tool merges these third-party scores into the model-index YAML while maintaining source attribution for accurate leaderboard tracking.

Can I run lighteval and vLLM evaluations to update Hugging Face model metadata?▼

You need a Hugging Face write token to push validated model-index YAML entries to the repository. An optional Artificial Analysis API key is required if you want to fetch and import third-party benchmark scores from their platform.

Does this tool validate model-index YAML before creating a pull request on Hugging Face?▼

Yes, the tool validates model-index YAML, merges it with existing entries, and checks for open pull requests before creating new ones. This safe publishing workflow prevents spamming repositories and ensures metadata accuracy.

What is the best way to extract evaluation tables from a model README for the Hugging Face model-index?▼

The best way to extract evaluation tables from a model README is to parse the markdown tables and convert them into structured model-index YAML entries. This automated extraction eliminates manual formatting errors and prepares the data for immediate publishing.

Related Skills

Explore More Software Engineering Skills→
huggingfacehuggingface
10.9k

hugging-face-evaluation

Convert README evaluation tables into Hugging Face model-index format.

Official
Advanced
HouseGarofaloHouseGarofalo
3

storybook

Document and test React UI components with Storybook stories and MDX.

Official
Intermediate
huggingfacehuggingface
10.9k

huggingface-community-evals

Evaluate Hugging Face Hub models locally with inspect-ai and lighteval.

Official
Intermediate
cpich3gcpich3g

hugging-face-evaluation

Extract evaluation tables from READMEs and update Hugging Face model-index metadata.

Community
Advanced