huggingface-community-evals

You can evaluate Hugging Face Hub models locally by running inspect-ai and lighteval directly on your GPU. This Skill configures the local environment to assess model performance using your own hardware.

The best way to choose between vLLM, Transformers, and accelerate is to use this Skill's backend selection feature. It allows you to run local GPU evaluations and compare model performance across these three distinct inference backends.

Yes, you can use inspect-ai with lighteval for local GPU evaluations. This Skill integrates both tools to provide a local evaluation environment for assessing Hugging Face Hub models on your own hardware.

You do not need vLLM to run local evaluations with Hugging Face Transformers. This Skill supports backend selection, allowing you to choose between vLLM, Hugging Face Transformers, and accelerate based on your specific evaluation needs.

Yes, this local model evaluation approach works with models like meta-llama/Llama-3.2-1B. You can quickly start running inspect-ai evaluations with local GPU inference to assess its performance.