skills.rest
patchy631
patchy631COMMUNITY
Explore all skills by this author →
𝕏@akshay_pachaarGitHub

hugging-face-evaluation

Publish and update evaluation results on Hugging Face model cards.

36.9k|6.1k|Updated Oct 21, 2024
Search Tags:#automation#evaluation#benchmark#HuggingFace#model-index#table-extraction
One-click install
npx skills add https://github.com/patchy631/ai-engineering-hub --skill hugging-face-evaluation-patchy631
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: hugging-face-evaluation
Source: https://github.com/patchy631/ai-engineering-hub/tree/main/hugging-face-skills/skills/hugging-face-evaluation
Command: npx skills add https://github.com/patchy631/ai-engineering-hub --skill hugging-face-evaluation-patchy631

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires huggingface-hub, markdown-it-py, python-dotenv, pyyaml, requests, and includes scripts (resource) components.

What problem does it solve?

This Skill streamlines the process of adding and maintaining evaluation results on Hugging Face model cards, enabling teams to track benchmarks, import external scores, and publish updates via PR workflows.

Core Features & Use Cases

  • Extract evaluation tables from README content and convert them into model-index entries.
  • Import benchmark scores from Artificial Analysis API and merge with existing results.
  • Run custom model evaluations using vLLM or HuggingFace Inference backends for up-to-date benchmarks.
  • Automatically generate model-index YAML compatible with model cards and leaderboard integrations.
  • Use cases: a team updating a model card after a new evaluation run, or a researcher publishing results with traceable sources.

Quick Start

To start, clone this skill directory into your project and run:

  • Preview evaluation extraction: uv run scripts/evaluation_manager.py extract-readme --repo-id "your-username/your-model" --dry-run
  • Import AA scores: AA_API_KEY="..." uv run scripts/evaluation_manager.py import-aa --creator-slug "your-organization" --model-name "your-model" --repo-id "your-username/your-model" --create-pr
  • Run a local vLLM evaluation: uv run scripts/lighteval_vllm_uv.py --model "model-id" --tasks "leaderboard|mmlu|5" --backend vllm

Frequently Asked Questions about hugging-face-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automatically update Hugging Face model cards with new evaluation results?▼

To update Hugging Face model cards with new evaluation results, you can automatically extract benchmark tables from README content and push updates via pull requests or direct commits. This streamlines tracking benchmarks and maintaining model-index YAML data.

How do I extract evaluation tables from a Hugging Face README and convert them to model-index YAML?▼

You can extract evaluation tables from a Hugging Face README and convert them to model-index YAML entries using the extract-readme command. Running this in dry-run mode previews the extracted data before applying actual changes to the model card.

Can I import benchmark scores from the Artificial Analysis API into my Hugging Face model card?▼

Yes, you can import benchmark scores from the Artificial Analysis API and merge them with existing results. You need to provide your AA_API_KEY and target repository details to generate model-index YAML and push updates via pull requests.

How do I run custom model evaluations using vLLM for Hugging Face benchmarks?▼

You can run custom model evaluations using vLLM by executing the lighteval script with your specified model ID and tasks. This generates up-to-date benchmark scores using the vLLM or HuggingFace Inference backends.

What is the best way to manage Hugging Face model evaluations for an ML team?▼

The best way to manage Hugging Face model evaluations is to automate publishing and updating results on model cards. This approach integrates external benchmark scores, tracks evaluation history, and ensures model-index compatibility with leaderboards.

Does publishing evaluation results to Hugging Face model cards require direct write access?▼

Publishing evaluation results to Hugging Face model cards does not strictly require direct write access. You can configure the workflow to submit updates via pull requests, allowing repository maintainers to review and merge benchmark changes.

Related Skills

Explore More Data & Analytics Skills→
erenisiklarerenisiklar

hugging-face-evaluation

Extract and update Hugging Face model card evaluation results from README tables and benchmarks.

Community
Advanced
POWERFULMOVESPOWERFULMOVES
1

hugging-face-evaluation-manager

Extract evaluation tables from READMEs and import benchmark scores from the Artificial Analysis API.

Community
Advanced
BlackRoad-OS-IncBlackRoad-OS-Inc
1

hugging-face-evaluation

Consolidate README benchmarks and Artificial Analysis imports into Hugging Face model-index entries.

Official
Advanced
BennooBennoo
4

hugging-face-evaluation

Automate Hugging Face model card updates with evaluation results.

Community
Advanced