lmms-eval

Run `npx skills add EvolvingLMMs-Lab/lmms-eval --all -g -y` in your terminal to install all tools in this suite globally.

It is a unified framework for benchmarking large multimodal models on image, video, and audio tasks. It gives reproducible, statistically trustworthy scores across 100+ benchmarks and 30+ model backends.

Run `python -m lmms_eval --model qwen2_5_vl --model_args pretrained=Qwen/Qwen2.5-VL-3B-Instruct --tasks mme --batch_size 1 --limit 8`. If metrics print, your setup works.

Yes. The built-in HTTP eval server accepts async job submissions from your training loop, and you collect results later by job ID.

It supports 30+ backends including Qwen-VL, LLaVA, vLLM, SGLang, and OpenAI-compatible APIs, with chat-template models recommended for new integrations.