skills.rest
BlackRoad-OS-Inc
BlackRoad-OS-IncOFFICIAL
Explore all skills by this author →
WebsiteGitHub

hugging-face-evaluation

Consolidate README benchmarks and Artificial Analysis imports into Hugging Face model-index entries.

1|Updated Feb 20, 2026
Search Tags:#automation#integration#evaluation#benchmark#huggingface#model-index#model-card
One-click install
npx skills add https://github.com/BlackRoad-OS-Inc/blackroad-operator --skill hugging-face-evaluation-blackroad-os-inc
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: hugging-face-evaluation
Source: https://github.com/BlackRoad-OS-Inc/blackroad-operator/tree/main/agents/skills/skills/hugging-face-evaluation
Command: npx skills add https://github.com/BlackRoad-OS-Inc/blackroad-operator --skill hugging-face-evaluation-blackroad-os-inc

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires huggingface-hub>=0.26.0, markdown-it-py>=3.0.0, python-dotenv>=1.2.1, pyyaml>=6.0.3, requests>=2.32.5, inspect-ai>=0.3.0, inspect-evals, openai, lighteval[accelerate,vllm]>=0.6.0, vllm>=0.4.0, torch>=2.0.0, transformers>=4.40.0, accelerate>=0.30.0, and includes scripts (resource) components.

What problem does it solve?

Add and manage evaluation results on Hugging Face model cards by consolidating README benchmarks, Artificial Analysis imports, and custom evaluations into model-index metadata.

Core Features & Use Cases

  • Extract evaluation tables from README content and convert them into model-index entries.
  • Import benchmark scores from Artificial Analysis and merge them with existing evaluations.
  • Run custom model evaluations with vLLM or accelerate backends (via lighteval, inspect-ai) on local hardware or HF Jobs.
  • Update model cards, ensuring source attribution and compatibility with Papers with Code metadata.

Quick Start

Run the integrated workflow to attach evaluations to a model card by extracting from a README or importing Artificial Analysis data.

Frequently Asked Questions about hugging-face-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I add evaluation results to a Hugging Face model card?▼

To update a Hugging Face model card with evaluations, this Skill consolidates README benchmarks, Artificial Analysis imports, and custom evaluation results into model-index metadata entries. It ensures source attribution and Papers with Code compatibility.

Can I run custom model evaluations with vLLM before updating the model card?▼

Yes, you can run custom model evaluations using vLLM or accelerate backends via lighteval and inspect-ai on local hardware or HF Jobs before updating the model card with the results.

How do I import benchmark scores from Artificial Analysis into Hugging Face metadata?▼

You can import benchmark scores from Artificial Analysis by providing your AA_API_KEY, allowing the Skill to fetch and merge those scores with existing evaluations into the Hugging Face model-index.

Do I need an HF_TOKEN to automate attaching evaluations to model cards?▼

Yes, an HF_TOKEN is required for full functionality to authenticate and push updates to Hugging Face model cards, alongside an AA_API_KEY if importing scores from Artificial Analysis.

What is the best way to extract evaluation tables from a README and convert them to model-index entries?▼

The best way to extract evaluation tables from README content is using the integrated markdown-it-py parsing workflow, which converts benchmark tables into structured Hugging Face model-index entries automatically.

Can I use lighteval with accelerate backends for Hugging Face model evaluations?▼

Yes, lighteval supports both vLLM and accelerate backends for running custom model evaluations, allowing you to execute benchmarks on local hardware or HF Jobs before updating the model card.

Related Skills

Explore More Software Engineering Skills→
oabdelmaksoudoabdelmaksoud
4

hugging-face-evaluation

Update Hugging Face model cards with evaluation results from README tables and workflows.

Community
Advanced
3kpro3kpro

hugging-face-evaluation

Manage evaluation results in Hugging Face model cards via READMEs, Artificial Analysis, vLLM, or lighteval.

Community
Advanced
huggingfacehuggingface
10.9k

huggingface-community-evals

Evaluate Hugging Face Hub models locally with inspect-ai and lighteval.

Official
Intermediate
FacuMFacuM

hugging-face-evaluation

Extract evaluation tables from Hugging Face model README files.

Community
Advanced