hugging-face-evaluation

Consolidate README benchmarks and Artificial Analysis imports into Hugging Face model-index entries.

1|Updated Feb 20, 2026

Search Tags:#automation #integration #evaluation #benchmark #huggingface #model-index #model-card

Ready to run

npx skills add https://github.com/BlackRoad-OS-Inc/blackroad-operator --skill hugging-face-evaluation-blackroad-os-inc

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: hugging-face-evaluation
Source: https://github.com/BlackRoad-OS-Inc/blackroad-operator/tree/main/agents/skills/skills/hugging-face-evaluation
Command: npx skills add https://github.com/BlackRoad-OS-Inc/blackroad-operator --skill hugging-face-evaluation-blackroad-os-inc

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires huggingface-hub>=0.26.0, markdown-it-py>=3.0.0, python-dotenv>=1.2.1, pyyaml>=6.0.3, requests>=2.32.5, inspect-ai>=0.3.0, inspect-evals, openai, lighteval[accelerate,vllm]>=0.6.0, vllm>=0.4.0, torch>=2.0.0, transformers>=4.40.0, accelerate>=0.30.0, and includes scripts (resource) components.

What problem does it solve?

Add and manage evaluation results on Hugging Face model cards by consolidating README benchmarks, Artificial Analysis imports, and custom evaluations into model-index metadata.

Core Features & Use Cases

Extract evaluation tables from README content and convert them into model-index entries.
Import benchmark scores from Artificial Analysis and merge them with existing evaluations.
Run custom model evaluations with vLLM or accelerate backends (via lighteval, inspect-ai) on local hardware or HF Jobs.
Update model cards, ensuring source attribution and compatibility with Papers with Code metadata.

Quick Start

Run the integrated workflow to attach evaluations to a model card by extracting from a README or importing Artificial Analysis data.

Frequently Asked Questions about hugging-face-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I add evaluation results to a Hugging Face model card?▼

To update a Hugging Face model card with evaluations, this Skill consolidates README benchmarks, Artificial Analysis imports, and custom evaluation results into model-index metadata entries. It ensures source attribution and Papers with Code compatibility.

Can I run custom model evaluations with vLLM before updating the model card?▼

Yes, you can run custom model evaluations using vLLM or accelerate backends via lighteval and inspect-ai on local hardware or HF Jobs before updating the model card with the results.

How do I import benchmark scores from Artificial Analysis into Hugging Face metadata?▼

You can import benchmark scores from Artificial Analysis by providing your AA_API_KEY, allowing the Skill to fetch and merge those scores with existing evaluations into the Hugging Face model-index.

Do I need an HF_TOKEN to automate attaching evaluations to model cards?▼

Yes, an HF_TOKEN is required for full functionality to authenticate and push updates to Hugging Face model cards, alongside an AA_API_KEY if importing scores from Artificial Analysis.

What is the best way to extract evaluation tables from a README and convert them to model-index entries?▼

The best way to extract evaluation tables from README content is using the integrated markdown-it-py parsing workflow, which converts benchmark tables into structured Hugging Face model-index entries automatically.

Can I use lighteval with accelerate backends for Hugging Face model evaluations?▼

Yes, lighteval supports both vLLM and accelerate backends for running custom model evaluations, allowing you to execute benchmarks on local hardware or HF Jobs before updating the model card.