huggingface-community-evals

To evaluate Hugging Face models locally, use this Skill to run inspect-ai and lighteval across vLLM or Transformers backends on your GPU or CPU. It handles backend selection, smoke tests, and scenario comparisons without remote HF Jobs orchestration.

Yes, you can run lighteval evaluations using a vLLM backend on a local GPU. The Skill provides backend selection guidance specifically for local GPU and CPU environments, allowing flexible execution of Hugging Face Hub model evaluations.

Yes, you need an HF_TOKEN to evaluate gated Hugging Face models locally. The Skill requires a local Python environment configured with inspect-ai, lighteval, vLLM or Transformers, and the token to authenticate and access restricted models.

The best way to compare vLLM and Transformers backends is using this Skill's scenario comparison features. It applies inspect-ai and lighteval to test Hugging Face models across different backends, highlighting performance differences directly in your local environment.

Local model evaluation runs directly on your hardware using inspect-ai and lighteval, whereas the hugging-face-jobs remote workflow handles cloud orchestration. This Skill focuses on local testing but can hand off to remote execution when needed.

You need dependencies like inspect-ai, torch, and vLLM to evaluate Hugging Face models because they provide the underlying execution frameworks and hardware acceleration required to run backend operations, model loading, and local evaluation tasks properly.