huggingface-community-evals

Evaluate Hugging Face models locally by running ready-to-use scripts that execute inspect-ai and lighteval against local hardware. The workflow automates backend selection between vLLM, Transformers, and accelerate, validating token handling and model reliability in isolated environments.

Comparing vLLM, Transformers, and accelerate backends is handled through side-by-side backend comparison scripts. This enables hardware-aware decision-making for local GPU setups by running smoke tests and validating model performance across different inference engines.

Local GPU evaluation requires dependencies including inspect-ai, inspect-evals, vLLM, torch, transformers, lighteval, and accelerate. These packages provide the necessary backend support for running isolated model evaluations and smoke tests on local hardware.

Lighteval works with vLLM as a supported backend for local Hugging Face Hub model testing. The workflow includes smoke tests and backend selection logic to validate token handling and model reliability using local GPU resources.

Remote compute is appropriate when local GPU resources are insufficient for larger model evaluations. This local evaluation workflow provides clear handoff guidance to other skills when remote compute or jobs orchestration becomes necessary for scaling.