skills.rest
sitammeur
sitammeurCOMMUNITY
Explore all skills by this author →
Website𝕏@sitammeurGitHub

hugging-face-evaluation

Merge Hugging Face model card evaluation data from READMEs and Artificial Analysis.

1|Updated Jan 22, 2026
Search Tags:#huggingface#model-index#vllm#evaluate#lighteval#readme-tables#artificial-analysis
One-click install
npx skills add https://github.com/sitammeur/ml-agent-skills --skill hugging-face-evaluation-sitammeur
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: hugging-face-evaluation
Source: https://github.com/sitammeur/ml-agent-skills/tree/main/skills/hugging-face-evaluation
Command: npx skills add https://github.com/sitammeur/ml-agent-skills --skill hugging-face-evaluation-sitammeur

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires huggingface-hub, markdown-it-py, python-dotenv, pyyaml, requests, inspect-ai, inspect-evals, openai, lighteval[accelerate,vllm], torch, transformers, accelerate, vllm, and includes scripts (resource) components.

What problem does it solve?

This Skill helps data teams automatically manage and publish evaluation results on Hugging Face model cards, consolidating scattered benchmarks into a single, up-to-date source of truth.

Core Features & Use Cases

  • Extract evaluation tables from READMEs and convert them to model-index format.
  • Import benchmarks from Artificial Analysis and merge them with existing data without overwriting.
  • Run custom model evaluations with vLLM or lighteval in Hugging Face Jobs for scalable testing.
  • Works with the model-index metadata specification to display scores on HF model hubs.

Quick Start

Use uv run to extract, import AA data, and view results. For example:

  • uv run scripts/evaluation_manager.py extract-readme --repo-id "your-username/your-model"
  • AA_API_KEY=your-aa-key uv run scripts/evaluation_manager.py import-aa --creator-slug "anthropic" --model-name "claude-sonnet-4" --repo-id "anthropic/claude-sonnet-4" --create-pr
  • uv run scripts/evaluation_manager.py show --repo-id "your-username/your-model"

Frequently Asked Questions about hugging-face-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I update Hugging Face model card evaluation tables without overwriting existing data?▼

To update Hugging Face model card evaluation tables without overwriting existing data, this skill extracts tables from READMEs and merges imported benchmarks using the model-index spec. It safely consolidates new scores while preserving your current model metadata.

How do I import Artificial Analysis benchmarks into a Hugging Face model card?▼

You can import Artificial Analysis benchmarks into a Hugging Face model card by using the import-aa script command with your AA API key. This fetches external benchmark scores and merges them into your existing model-index metadata.

Can I run custom model evaluations with vLLM and lighteval in Hugging Face Jobs?▼

Yes, you can run custom model evaluations with vLLM and lighteval in Hugging Face Jobs. This skill provides integrated scripts to execute scalable testing and automatically consolidate the resulting evaluation metrics back into your model card.

Does this skill require Hugging Face authentication to manage model metadata?▼

Yes, Hugging Face authentication is required to manage model metadata. The skill leverages HF authentication via the huggingface-hub dependency to securely access repositories, extract README tables, and publish consolidated evaluation results.

What is the best way to consolidate scattered benchmarks into a single Hugging Face model card?▼

The best way to consolidate scattered benchmarks into a single Hugging Face model card is using the model-index metadata specification. This skill standardizes extracted README tables and Artificial Analysis imports into one unified source of truth.

Why does my imported evaluation data not show up on the Hugging Face model hub?▼

Imported evaluation data may not show up on the Hugging Face model hub if it is not formatted according to the model-index specification. This skill ensures your consolidated benchmarks are correctly structured for proper display on the hub.

Related Skills

Explore More Data & Analytics Skills→
pingqLINpingqLIN

hugging-face-evaluation

Extract README evaluation tables and import Artificial Analysis benchmarks into Hugging Face model cards.

Community
Advanced
ISAQQSAIISAQQSAI

hugging-face-evaluation-manager

Automate evaluation result updates in Hugging Face model cards.

Community
Advanced
mashharukimashharuki

hugging-face-evaluation

Extract evaluation tables from Hugging Face model README files.

Community
Advanced
Andrew-GirgisAndrew-Girgis

huggingface-community-evals

Evaluate Hugging Face Hub models locally with inspect-ai and lighteval.

Community
Advanced