huggingface-community-evals

Run local Hugging Face model evaluations by configuring inference backends like vLLM or Transformers with benchmark frameworks such as inspect-ai and lighteval. You can execute smoke tests first, then scale to larger local GPU runs for benchmarks like MMLU or GSM8K.

Evaluating LLMs locally with vLLM and lighteval involves selecting an appropriate inference backend based on model compatibility and hardware. This approach automates framework configuration, enabling efficient benchmark execution directly on local GPU environments.

Yes, you can use inspect-ai with Hugging Face Transformers for local GPU inference. The Skill supports backend selection between vLLM, Transformers, and accelerate, allowing you to optimize evaluation workflows based on your available hardware and throughput needs.

Using accelerate is optional but supported for running local benchmarks on Hugging Face models. Backend selection logic chooses between vLLM, Transformers, and accelerate based on your specific model compatibility, throughput requirements, and available local GPU hardware.

Backend selection for local model evaluation works by evaluating model compatibility, throughput needs, and available hardware to choose between vLLM, Hugging Face Transformers, and accelerate. This hardware-aware execution control ensures optimal performance during benchmark runs.

Limitations when running inspect-ai benchmarks on local GPU environments depend on your selected inference backend and hardware capacity. The Skill requires model access credentials, evaluation scripts, and sufficient local GPU resources to scale from smoke tests to larger runs.