skills.rest
Tyler-R-Kendrick
Tyler-R-KendrickCOMMUNITY
Explore all skills by this author →
GitHub

huggingface-community-evals

Executes Hugging Face Hub model evaluations locally using inspect-ai and light frameworks with vLLM, Transformers, or accelerate backends.

1|Updated Jul 12, 2026
Search Tags:#benchmarking#evaluation#inspect-ai#huggingface#vllm#lighteval
One-click install
npx skills add https://github.com/Tyler-R-Kendrick/slm-training --skill huggingface-community-evals-tyler-r-kendrick
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: huggingface-community-evals
Source: https://github.com/Tyler-R-Kendrick/slm-training/tree/main/.agents/skills/huggingface-community-evals
Command: npx skills add https://github.com/Tyler-R-Kendrick/slm-training --skill huggingface-community-evals-tyler-r-kendrick

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires inspect-ai, inspect-evals, openai, vllm, torch, transformers, lighteval, accelerate, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This skill solves the challenge of running standardized model evaluations on local hardware, allowing developers to benchmark Hugging Face models without relying on remote job orchestration or external API providers.

Core Features & Use Cases

  • Local Evaluation: Run inspect-ai and lighteval benchmarks directly on your local GPU or CPU.
  • Backend Flexibility: Seamlessly switch between vLLM for high-throughput inference, Hugging Face Transformers for compatibility, or accelerate for distributed setups.
  • Workflow Integration: Provides a clear boundary for local testing before handing off to remote Hugging Face Jobs for large-scale production runs.

Quick Start

Run the huggingface-community-evals skill to evaluate the meta-llama/Llama-3.2-1B model on the mmlu task using the local vLLM backend.

Frequently Asked Questions about huggingface-community-evals

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run local model evaluations for Hugging Face Hub models?▼

Run local model evaluations by executing this skill to benchmark Hugging Face Hub models using inspect-ai and lighteval frameworks directly on your local hardware. It supports vLLM, Transformers, and accelerate backends.

Can I benchmark Hugging Face models with vLLM on my local GPU?▼

Yes, you can benchmark Hugging Face models with vLLM on your local GPU. This skill supports diverse inference backends including vLLM for high-throughput inference, alongside Transformers and accelerate for distributed setups.

Do I need uv to run inspect-ai and lighteval benchmarks?▼

Yes, you need uv to run inspect-ai and lighteval benchmarks. The skill requires uv for environment management and local hardware access to perform smoke tests and full-scale model validation.

What is the best way to test Hugging Face models locally before remote job orchestration?▼

The best way to test Hugging Face models locally before remote job orchestration is using this skill. It provides a clear boundary for local testing before handing off to remote Hugging Face Jobs for large-scale production runs.

Does local benchmarking with lighteval support distributed setups using accelerate?▼

Yes, local benchmarking with lighteval supports distributed setups using accelerate. The skill seamlessly switches between vLLM, Hugging Face Transformers for compatibility, or accelerate for distributed hardware configurations.

Related Skills

Explore More Software Engineering Skills→
niznik-devniznik-dev
16

create-inspect-task

Create custom inspect-ai evaluation tasks with experiment-guided and standalone modes.

Community
Intermediate
AnushkaTandon26AnushkaTandon26

hugging-face-evaluation

Attach evaluation results to Hugging Face model cards as structured model-index data.

Community
Intermediate
iani-kuliiani-kuli

huggingface-community-evals

Execute local Hugging Face model evaluations with inspect-ai or lighteval frameworks.

Community
Intermediate
RheasilviaRheasilvia

huggingface-hub

Manage Hugging Face Hub repositories and sync large files.

Community
Intermediate