quark-torch-llm-eval

You can run LLM accuracy benchmarks on AMD ROCm by orchestrating model inspection, backend serving, and benchmark execution. This Skill automates evaluating models across tasks like gsm8k and mmlu using vLLM, SGLang, or ATOM serving with structured reporting.

Yes, vLLM and SGLang work with AMD ROCm for LLM evaluation by providing OpenAI-compatible serving. The Skill validates readiness with smoke tests before executing accuracy checks using lm-eval or lighteval frameworks.

The best way to evaluate quantized LLM models on ROCm is using an automated pipeline that handles backend serving and smoke testing. This Skill applies to quantized and base models, executing benchmarks and detecting reference-score anomalies for diagnostics.

Yes, you can compare LLM benchmark results against reference scores on AMD GPUs. The Skill generates structured reports with reference-score anomaly detection to guide troubleshooting when accuracy issues appear.

LLM evaluation smoke tests may fail on ROCm due to backend serving issues or improper ROCm-aware GPU selection. The Skill validates readiness with smoke tests and provides diagnostics to guide troubleshooting when launch issues appear.

Yes, you need OpenAI-compatible serving to run mmlu_pro and gpqa evaluations. The Skill requires OpenAI-compatible serving from vLLM, SGLang, or ATOM to execute benchmarks using lm-eval or lighteval frameworks.