skills.rest
Dianshu-Liao
Dianshu-LiaoCOMMUNITY
Explore all skills by this author →
GitHub

hugging-face-evaluation-manager

Extract README evaluation tables into Hugging Face model-index YAML.

2|Updated May 20, 2026
Search Tags:#benchmarking#model-index#hf-jobs#hugging-face#artificial-analysis#readme parsing#evaluation automation
One-click install
npx skills add https://github.com/Dianshu-Liao/SkilLGuard --skill hugging-face-evaluation-manager-dianshu-liao
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: hugging-face-evaluation-manager
Source: https://github.com/Dianshu-Liao/SkilLGuard/tree/main/data/skills/hf_model_evaluation
Command: npx skills add https://github.com/Dianshu-Liao/SkilLGuard --skill hugging-face-evaluation-manager-dianshu-liao

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires huggingface-hub, markdown-it-py, python-dotenv, pyyaml, requests, and includes scripts (resource) components.

What problem does it solve?

Updating Hugging Face model cards with accurate, structured benchmark results is time-consuming, error-prone, and often requires manual formatting and repository-specific work.

Core Features & Use Cases

  • Extract eval tables from README into model-index YAML for leaderboard/metadata compatibility.
  • Import scores from Artificial Analysis and merge them into an existing model card.
  • Run evaluations on HF Jobs and custom GPUs using inspect-ai with inference providers or lighteval with vLLM/accelerate backends.

Quick Start

Run table extraction from an existing model README by telling the AI to execute: uv run scripts/evaluation_manager.py extract-readme --repo-id "username/model-name" --table 1 --apply

Frequently Asked Questions about hugging-face-evaluation-manager

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I publish benchmark results to a Hugging Face model card?▼

To publish benchmark results to a Hugging Face model card, this tool extracts evaluation metrics from README markdown tables or the Artificial Analysis API and formats them as model-index YAML metadata. It can apply changes directly or create a pull request.

How do I extract markdown tables from a README into model-index YAML?▼

You can extract markdown tables into model-index YAML by running the extraction script with a repository ID and target table number. It parses the README using markdown-it-py and formats the output as leaderboard-ready metadata.

Can I run model evaluations on HF Jobs using inspect-ai or lighteval?▼

Yes, you can run model evaluations on HF Jobs or local GPU hardware using inspect-ai with inference providers or lighteval with vLLM and accelerate backends. The resulting scores are then imported into the model card.

Do I need an HF_TOKEN to update Hugging Face model-index metadata?▼

Yes, you need an authenticated HF_TOKEN to update Hugging Face model-index metadata and apply changes to the repository. An AA_API_KEY is also required if ingesting external benchmark scores from the Artificial Analysis API.

What is the best way to import Artificial Analysis scores into a Hugging Face model card?▼

The best way to import Artificial Analysis scores is using the evaluation manager to fetch data via the AA_API_KEY and merge the benchmark results directly into your existing Hugging Face model card as structured YAML metadata.

Related Skills

Explore More Software Engineering Skills→
BlackRoad-OS-IncBlackRoad-OS-Inc
1

hugging-face-evaluation

Consolidate README benchmarks and Artificial Analysis imports into Hugging Face model-index entries.

Official
Advanced
huggingfacehuggingface
10.9k

hugging-face-evaluation

Convert README evaluation tables into Hugging Face model-index format.

Official
Advanced
tripplen23tripplen23
1

huggingface-community-evals

Evaluate Hugging Face Hub models locally with inspect-ai and lighteval.

Community
Advanced
cpich3gcpich3g

hugging-face-evaluation

Extract evaluation tables from READMEs and update Hugging Face model-index metadata.

Community
Advanced