skills.rest
cloudrift-ai
cloudrift-aiOFFICIAL
Explore all skills by this author →
GitHub

benchmark-new-model

Automate onboarding of HuggingFace models into emmy with deployment validation and benchmarking.

75|8|Updated Aug 2, 2025
Search Tags:#vllm#sglang#model onboarding#llm benchmarking#gpu deployment#inference validation#recipe automation
One-click install
npx skills add https://github.com/cloudrift-ai/emmy --skill benchmark-new-model
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: benchmark-new-model
Source: https://github.com/cloudrift-ai/emmy/tree/main/.claude/skills/benchmark-new-model
Command: npx skills add https://github.com/cloudrift-ai/emmy --skill benchmark-new-model

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill eliminates the tedious, error-prone manual work of onboarding previously unsupported LLMs into emmy, validating their deployment, and running reliable performance benchmarks, saving you hours of wasted GPU time and failed test runs.

Core Features & Use Cases

  • Automated Recipe Creation: Researches optimal engine versions, Docker images, and parser configurations for new HuggingFace models to generate ready-to-use deployment recipes.
  • Pre-Benchmark Validation: Verifies the model launches correctly, serves valid responses, and supports its claimed context length before running costly benchmarks to avoid wasting resources on broken configs.
  • Time-Capped Benchmarking: Runs performance tests with a strict 20-minute wall-clock limit, automatic error handling, and reproducible metrics to get clear throughput and latency results.
  • Use Case: If you want to test the performance of a new Qwen3 MoE model on your CloudRift GPU server, this Skill will automatically find the right vLLM image, configure tool and reasoning parsers, validate the deployment, and run a capped benchmark to deliver actionable performance data.

Quick Start

Use the benchmark-new-model skill to onboard the HuggingFace model Qwen/Qwen3-8B-Instruct and run a performance benchmark on your remote GPU server.

Frequently Asked Questions about benchmark-new-model

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I onboard a new HuggingFace model for vLLM or SGLang deployment?▼

To onboard a new HuggingFace model, this Skill automates recipe creation by researching optimal vLLM or SGLang engine versions, Docker images, and parser configurations, then validates the deployment before benchmarking.

Can I benchmark quantized, MoE, or multimodal LLM architectures on a remote GPU server?▼

Yes, you can benchmark quantized, MoE, and multimodal LLM architectures on remote GPU servers. The Skill handles engine image pinning and parser configuration to ensure reproducible inference validation for these complex model types.

What is the best way to ensure reproducible LLM benchmarking results and avoid configuration drift?▼

The best way to ensure reproducible LLM benchmarking results is by pinning vLLM or SGLang engine image versions and automating tool and reasoning parser configuration, which prevents configuration drift across benchmark runs.

Does vLLM benchmarking support automatic error handling and strict time limits?▼

Yes, vLLM benchmarking supports automatic error handling and a strict 20-minute wall-clock time cap. This ensures you get clear throughput and latency results without wasting GPU time on stalled or broken configurations.

Why should I validate context length support before running a full LLM performance benchmark?▼

Validating context length support before running a full LLM performance benchmark prevents wasting compute resources on broken configurations. Pre-benchmark validation confirms the model launches correctly and serves valid responses prior to costly testing.

Related Skills

Explore More Software Engineering Skills→
hugoguhugogu

docker-deploy

Automate multi-environment Docker deployment with docker-compose and registry prefix switching.

Community
Advanced
dejavoo21dejavoo21

worker-benchmarks

Benchmark worker system performance across detection, registry, selection, caching, and concurrency workloads.

Community
Advanced
vasilyu1983vasilyu1983
73

ai-llm-inference

Optimize LLM inference performance and cost with vLLM and quantization.

Community
Advanced
Mattyreed1Mattyreed1

mcp-builder

Guide Python or TypeScript development of Model Context Protocol servers with evaluation workflows.

Community
Advanced