skills.rest
Nymbo
NymboCOMMUNITY
Explore all skills by this author →
𝕏@nymboqtGitHub

hugging-face-evaluation-manager

Extract README evaluation tables and import Artificial Analysis scores into Hugging Face model cards.

Updated Aug 23, 2026
Search Tags:#evaluation#vLLM#huggingface#Artificial-Analysis#readme-extraction#model-index
One-click install
npx skills add https://github.com/Nymbo/Skills --skill hugging-face-evaluation-manager
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: hugging-face-evaluation-manager
Source: https://github.com/Nymbo/Skills/tree/main/hugging-face-evaluation-manager
Command: npx skills add https://github.com/Nymbo/Skills --skill hugging-face-evaluation-manager

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires huggingface_hub, markdown-it-py, python-dotenv, pyyaml, requests, and includes scripts (resource) components.

What problem does it solve?

Helps update model cards with evaluation results by extracting from README content, importing benchmark scores from Artificial Analysis, and running custom model evaluations with vLLM or lighteval. Works with the model-index metadata format.

Core Features & Use Cases

  • Extract from README: Parse and convert evaluation tables in READMEs to model-index YAML.
  • Import AA scores: Pull benchmark data from Artificial Analysis API and merge with existing results.
  • Run evaluations: Execute vLLM or lighteval evaluations via local GPU or HF Jobs, with PR handling and validation.
  • Model-index updates: Merge results into model cards using the model-index format and ensure Papers with Code compatibility.

Quick Start

  • Preview extraction: uv run scripts/evaluation_manager.py extract-readme --repo-id "your-username/your-model" --dry-run
  • Apply extraction: uv run scripts/evaluation_manager.py extract-readme --repo-id "your-username/your-model"
  • Import AA scores: python scripts/evaluation_manager.py import-aa --creator-slug "anthropic" --model-name "claude-sonnet-4" --repo-id "username/model"

Frequently Asked Questions about hugging-face-evaluation-manager

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract evaluation results from Hugging Face model README tables into model-index format?▼

Extract evaluation results by parsing README tables with the evaluation manager, which converts structured table data into model-index YAML format. Run `uv run scripts/evaluation_manager.py extract-readme --repo-id "your-username/your-model"` to preview or apply extraction directly to your model card.

Can I import benchmark scores from Artificial Analysis into my model card?▼

Yes, import Artificial Analysis benchmark data using `python scripts/evaluation_manager.py import-aa --creator-slug "anthropic" --model-name "claude-sonnet-4" --repo-id "username/model"`. The tool fetches scores via the Artificial Analysis API and merges results into your existing model-index metadata.

What evaluation formats and table structures does the README extraction support?▼

The tool supports multiple README table formats and consolidates metrics from varied layouts. It uses markdown-it-py to parse tables and handles different column structures, extracting benchmark names, scores, and metadata for conversion to standardized model-index YAML entries.

Does the evaluation manager work with vLLM and lighteval evaluations?▼

Yes, the tool can run custom evaluations using vLLM or lighteval on local GPUs or via Hugging Face Jobs. It handles PR creation, model-index validation, and merges evaluation results back into model cards with full ecosystem compatibility.

What dependencies and setup do I need to use the evaluation manager?▼

Install huggingface_hub, markdown-it-py, python-dotenv, pyyaml, and requests. The tool requires access to Hugging Face repositories and optional Artificial Analysis API credentials. Use a `.env` file for authentication tokens and API keys.

How does the evaluation manager ensure model-index validation and Papers with Code compatibility?▼

The tool generates and merges model-index entries using validated YAML format, ensuring compatibility with Hugging Face model cards and Papers with Code integration. It validates structure and data attribution before creating pull requests to model repositories.

Related Skills

Explore More Data & Analytics Skills→
mashharukimashharuki

hugging-face-evaluation

Extract evaluation tables from Hugging Face model README files.

Community
Advanced
oabdelmaksoudoabdelmaksoud
4

hugging-face-paper-publisher

Index, publish, and manage academic papers on the Hugging Face Hub.

Community
Advanced
aisa-groupaisa-group
90

hugging-face-evaluation-manager

Manage structured evaluation results in Hugging Face model cards.

Official
Advanced
ISAQQSAIISAQQSAI

hugging-face-evaluation-manager

Automate evaluation result updates in Hugging Face model cards.

Community
Advanced