skills.rest
ClyptAI
ClyptAIOFFICIAL
Explore all skills by this author →
GitHub

huggingface-community-evals

Run local Hugging Face Hub model evaluations with inspect-ai and lighteval.

Updated Mar 8, 2026
Search Tags:#evaluation#gpu#local#inspect-ai#huggingface#backends#lighteval
One-click install
npx skills add https://github.com/ClyptAI/Clypt-Backend --skill huggingface-community-evals-clyptai
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: huggingface-community-evals
Source: https://github.com/ClyptAI/Clypt-Backend/tree/main/.agents/skills/hf-skills/skills/huggingface-community-evals
Command: npx skills add https://github.com/ClyptAI/Clypt-Backend --skill huggingface-community-evals-clyptai

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires inspect-ai, inspect-evals, openai, vllm, torch, transformers, accelerate, lighteval, and includes scripts (resource) components.

What problem does it solve?

Run evaluations for Hugging Face Hub models locally using inspect-ai and lighteval, enabling quick backend comparison and offline experimentation.

Core Features & Use Cases

  • Local evaluation of HF Hub models with inspect-ai and lighteval across vLLM, Transformers, and accelerate backends.
  • Supports smoke tests, backend selection, and performance comparisons for model selection.
  • Not intended for Hugging Face Jobs orchestration, model-card PRs, .eval_results publication, or community-evals automation.

Quick Start

Use uv run with either scripts/inspect_eval_uv.py or scripts/lighteval_vllm_uv.py to evaluate a Hugging Face Hub model on local hardware.

Frequently Asked Questions about huggingface-community-evals

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run local GPU evaluations on Hugging Face Hub models?▼

You can run local GPU evaluations on Hugging Face Hub models using provided scripts via uv run, executing either inspect_eval_uv.py or lighteval_vllm_uv.py to benchmark models offline on your local hardware.

What backends are supported for local model benchmarking with inspect-ai and lighteval?▼

Local model benchmarking supports vLLM, HuggingFace Transformers, and accelerate backends, enabling performance comparisons and backend selection to help you choose the optimal model deployment framework.

Can I use this to automate community evaluations and publish eval_results to model cards?▼

No, it is not intended for Hugging Face Jobs orchestration, model-card PRs, .eval_results publication, or community-evals automation; it strictly supports local offline experimentation and backend comparison.

Do I need vLLM and torch installed to run Hugging Face model smoke tests locally?▼

Yes, running local Hugging Face model smoke tests requires explicit dependencies including vllm, torch, transformers, accelerate, inspect-ai, inspect-evals, lighteval, and openai to successfully execute the evaluation scripts.

What is the best way to compare vLLM and Transformers performance for Hugging Face models?▼

The best way to compare vLLM and Transformers performance is running local evaluations across multiple backends using lighteval and inspect-ai scripts, which support backend selection and performance benchmarking for model selection.

Related Skills

Explore More Software Engineering Skills→
Tyler-R-KendrickTyler-R-Kendrick
1

huggingface-community-evals

Executes Hugging Face Hub model evaluations locally using inspect-ai and light frameworks with vLLM, Transformers, or accelerate backends.

Community
Intermediate
iani-kuliiani-kuli

huggingface-community-evals

Execute local Hugging Face model evaluations with inspect-ai or lighteval frameworks.

Community
Intermediate
huggingfacehuggingface
10.9k

huggingface-community-evals

Evaluate Hugging Face Hub models locally with inspect-ai and lighteval.

Official
Intermediate
huggingfacehuggingface
10.9k

huggingface-vision-trainer

Automate vision model training and fine-tuning on Hugging Face Jobs.

Official
Advanced