skills.rest
Cramer-69
Cramer-69COMMUNITY
Explore all skills by this author →
Website𝕏@cramerconsultGitHub

hugging-face-evaluation

Add and manage evaluation results in Hugging Face model cards using model-index metadata.

1|Updated Feb 18, 2026
Search Tags:#benchmark#model evaluation#model-index#vllm#lighteval#hugging face#model card
One-click install
npx skills add https://github.com/Cramer-69/skills-repository-3af2d1c7 --skill hugging-face-evaluation-cramer-69
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: hugging-face-evaluation
Source: https://github.com/Cramer-69/skills-repository-3af2d1c7/tree/main/packages/huggingface-skills/skills/hugging-face-evaluation
Command: npx skills add https://github.com/Cramer-69/skills-repository-3af2d1c7 --skill hugging-face-evaluation-cramer-69

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires huggingface-hub, markdown-it-py, python-dotenv, pyyaml, requests, inspect-ai, inspect-evals, openai, lighteval, vllm, torch, transformers, accelerate, and includes scripts (resource) components.

What problem does it solve? Hugging Face model cards often contain benchmark scores in README tables that are not exposed as structured model-index metadata, preventing leaderboard integration and evaluation widget display. This Skill extracts, imports, and publishes evaluation results to model cards without manual YAML editing. ## Core Features & Use Cases - README Table Extraction: Parse markdown evaluation tables with markdown-it-py, detect table formats (benchmarks as rows, columns, or transposed comparison tables), and convert them to model-index YAML. - Artificial Analysis Import: Fetch benchmark scores directly from the Artificial Analysis API and merge them into existing model cards with source attribution. - Custom Model Evaluation: Run evaluations with vLLM, lighteval, or inspect-ai locally on GPU or via Hugging Face Jobs, then publish results. - Use Case: A community contributor finds a model with benchmark tables in its README but no evaluation widget. They run get-prs to check for open PRs, use inspect-tables and extract-readme --create-pr to propose structured model-index metadata to the repository owner. ## Quick Start Ask the AI to inspect the evaluation tables in a Hugging Face model README and create a pull request adding the extracted scores as model-index metadata.

Frequently Asked Questions about hugging-face-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I add evaluation results to a Hugging Face model card?▼

Run the evaluation_manager.py script with the extract-readme command to parse benchmark tables from the model's README and convert them to model-index YAML. Use --apply to push directly or --create-pr to open a pull request for repositories you do not own.

How to import benchmark scores from Artificial Analysis to Hugging Face?▼

Set the AA_API_KEY environment variable and run the import-aa command with the creator slug, model name, and target repo ID. The script fetches scores from the Artificial Analysis API and formats them as model-index metadata with source attribution.

What is the difference between vLLM and lighteval for model evaluation?▼

vLLM is a high-performance GPU inference backend, while lighteval is HuggingFace's evaluation library that can use vLLM or accelerate as backends. The scripts support lighteval for Open LLM Leaderboard tasks and inspect-ai for UK AI Safety Institute benchmarks.

Can I run Hugging Face model evaluations without a local GPU?▼

Yes, you can submit evaluation jobs to Hugging Face Jobs infrastructure using hf jobs uv run with a hardware flavor like a10g-small. Inference provider evaluations via inspect-ai also run on provider infrastructure without local GPU requirements.

Why does table extraction fail on my model README?▼

Extraction fails when the README lacks markdown tables with numeric scores, when multiple evaluation tables exist without a --table selection, or when the model name does not exactly match a column or row. Run inspect-tables first to see table structure and available model names.

Should I check for existing pull requests before creating one?▼

Yes, always run the get-prs command before using --create-pr to avoid duplicate evaluation PRs. If open PRs exist, review their URLs and only proceed with explicit confirmation to prevent spamming model repository maintainers.

Related Skills

Explore More Software Engineering Skills→
sitammeursitammeur
1

hugging-face-evaluation

Merge Hugging Face model card evaluation data from READMEs and Artificial Analysis.

Community
Advanced
patronus-aipatronus-ai
2

hugging-face-evaluation-manager

Extract README evaluation tables into model-index YAML for Hugging Face model cards.

Official
Advanced
NymboNymbo

hugging-face-evaluation-manager

Extract README evaluation tables and import Artificial Analysis scores into Hugging Face model cards.

Community
Advanced
ISAQQSAIISAQQSAI

hugging-face-evaluation-manager

Automate evaluation result updates in Hugging Face model cards.

Community
Advanced