benchmark-new-model

What problem does it solve?

This Skill eliminates the tedious, error-prone manual work of onboarding previously unsupported LLMs into emmy, validating their deployment, and running reliable performance benchmarks, saving you hours of wasted GPU time and failed test runs.

Core Features & Use Cases

• Automated Recipe Creation: Researches optimal engine versions, Docker images, and parser configurations for new HuggingFace models to generate ready-to-use deployment recipes.
• Pre-Benchmark Validation: Verifies the model launches correctly, serves valid responses, and supports its claimed context length before running costly benchmarks to avoid wasting resources on broken configs.
• Time-Capped Benchmarking: Runs performance tests with a strict 20-minute wall-clock limit, automatic error handling, and reproducible metrics to get clear throughput and latency results.
• Use Case: If you want to test the performance of a new Qwen3 MoE model on your CloudRift GPU server, this Skill will automatically find the right vLLM image, configure tool and reasoning parsers, validate the deployment, and run a capped benchmark to deliver actionable performance data.

Quick Start

Use the benchmark-new-model skill to onboard the HuggingFace model Qwen/Qwen3-8B-Instruct and run a performance benchmark on your remote GPU server.