huggingface-community-evals

You can run local evaluations on Hugging Face models using inspect-ai by executing the provided deterministic scripts designed for smoke tests and task management without relying on HF Jobs orchestration.

Yes, you can evaluate private Hugging Face models locally by configuring lighteval and inspect-ai with flexible backend options, allowing you to easily select between vLLM, Transformers, and accelerate for your runs.

Yes, lighteval supports vLLM and Transformers backends for local model testing, enabling you to perform quick local smoke tests and compare backend performance directly through script-based execution.

The best way to compare vLLM and Transformers backends is by using the dedicated backend selection scripts provided, which allow configurable local evaluation runs to directly measure performance differences.

Yes, you need a GPU to run local Hugging Face model evaluations, as the dependencies require torch and accelerate to execute the models efficiently across the selected vLLM or Transformers backends.

No, you cannot publish results directly from these local scripts; they establish safe hand-off boundaries for remote job orchestration and publishing workflows to dedicated automation skills.