huggingface-community-evals

Run local model evaluations for Hugging Face Hub models with inspect-ai and lighteval.

2|Updated Mar 29, 2026

Search Tags:#llm #benchmarking #evaluation #inference #huggingface #vllm

Ready to run

npx skills add https://github.com/NagyVikt/codex-plugins --skill huggingface-community-evals-nagyvikt

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: huggingface-community-evals
Source: https://github.com/NagyVikt/codex-plugins/tree/main/plugins/hugging-face/skills/community-evals
Command: npx skills add https://github.com/NagyVikt/codex-plugins --skill huggingface-community-evals-nagyvikt

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires inspect-ai, inspect-evals, openai, vllm, torch, transformers, lighteval, accelerate, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This skill solves the complexity of setting up and running local model evaluations on Hugging Face Hub models, providing a standardized way to benchmark performance without needing to manage remote infrastructure.

Core Features & Use Cases

Local Evaluation: Execute inspect-ai and lighteval benchmarks directly on your local GPU hardware.
Backend Flexibility: Seamlessly switch between vLLM for high-throughput inference, Hugging Face Transformers, or accelerate for compatibility.
Workflow Integration: Easily perform smoke tests with sample limits before scaling to full evaluation runs.

Quick Start

Run the inspect-ai evaluation for the Llama-3.2-1B model on the mmlu task with a limit of 20 samples using the local script.

Frequently Asked Questions about huggingface-community-evals

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I run local LLM benchmarking for Hugging Face models on my own GPU?▼

Local LLM benchmarking for Hugging Face models is executed using inspect-ai and lighteval frameworks directly on your GPU hardware, enabling performance validation without managing remote infrastructure.

Can I use vLLM for high-throughput inference during local Hugging Face model evaluation?▼

Yes, you can use vLLM for high-throughput inference during local Hugging Face model evaluation. The skill supports backend flexibility, allowing seamless switching between vLLM, Hugging Face Transformers, and accelerate.

What is the best way to perform a smoke test before scaling to a full model evaluation run?▼

The best way to perform a smoke test before a full model evaluation run is to execute the benchmark with a sample limit. For example, running the inspect-ai evaluation on the mmlu task with a limit of 20 samples validates resource efficiency.

Does this local evaluation approach support backend fallback strategies when GPU resources are constrained?▼

Yes, this local evaluation approach supports backend fallback strategies for constrained GPU resources. You can switch from high-throughput vLLM inference to Hugging Face Transformers or accelerate for broader compatibility.

Why use inspect-ai and lighteval for local model performance validation instead of remote benchmarking platforms?▼

Using inspect-ai and lighteval for local model performance validation provides a standardized way to benchmark Hugging Face Hub models. This approach eliminates remote infrastructure management while supporting diverse local inference backends.