skills.rest
aws-samples
aws-samplesOFFICIAL
Explore all skills by this author →
WebsiteGitHub

bench-open-model

Benchmarks open-weight Hugging Face models on AWS EC2 GPU instances with vLLM.

1|Updated Aug 6, 2026
Search Tags:#aws#benchmark#gpu#throughput#ec2#vllm#hugging face
One-click install
npx skills add https://github.com/aws-samples/sample-gpu-open-weight-bench --skill bench-open-model-aws-samples
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: bench-open-model
Source: https://github.com/aws-samples/sample-gpu-open-weight-bench/tree/main/plugins/bench-open-model/skills/bench-open-model
Command: npx skills add https://github.com/aws-samples/sample-gpu-open-weight-bench --skill bench-open-model-aws-samples

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve? Choosing the right GPU instance and getting trustworthy throughput numbers for an open-weight model is hard: VRAM sizing is error-prone, inference caches contaminate benchmark results, and forgotten EC2 instances keep billing. This Skill automates the full cycle of sizing, provisioning, cache-honest load testing, reporting, and teardown on AWS. ## Core Features & Use Cases - VRAM sizing and instance selection: Computes weights, KV cache, and overhead from Hugging Face metadata, then ranks EC2 GPU instances using live pricing and availability, failing closed on architectures it cannot size reliably (MLA, sliding-window, Mamba/SSM). - Automated provisioning and teardown: Creates a single CloudFormation stack (instance, security group locked to your IP, key pair, IAM role) with AZ/region capacity fallback, and deletes it with verified teardown plus an orphan-stack sweep. - Cache-honest concurrency sweeps: Runs text or vision load tests with unique payloads, cold-cache restarts, and validation that rejects truncated or failed requests, so reported tokens-per-second and latency numbers are real. - Use Case: Ask Claude to benchmark a model like Qwen2.5-7B-Instruct; it picks a cost-appropriate GPU instance, serves the model on vLLM, sweeps concurrency levels, writes a report with the saturation point and cost-per-1k-tokens, then tears everything down. ## Quick Start Ask Claude to benchmark a specific Hugging Face model ID on AWS, for example by saying: benchmark the throughput and latency of meta-llama/Llama-3.1-8B-Instruct on AWS GPU hardware.

Frequently Asked Questions about bench-open-model

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I benchmark a Hugging Face model on AWS GPU instances?▼

Provide the Hugging Face model ID and the Skill sizes its VRAM needs, picks an EC2 GPU instance with live pricing, provisions it, serves the model on vLLM, runs a concurrency sweep, and tears everything down. You approve the plan and cost once before any spending begins.

How do I choose the right EC2 GPU instance for an LLM?▼

The sizing script computes weights plus KV cache plus overhead from the model's config.json and ranks candidate instances using live describe-instance-types data and On-Demand pricing. It prefers the smallest single-GPU instance that fits, since one replica per GPU beats tensor parallelism for models that fit on one card.

Does it support vision and multimodal models?▼

Yes, it handles text LLMs, MoE models, and multimodal vision/OCR models that vLLM can serve. Vision benchmarks use a --vision provisioning flag and generate unique page images at a chosen DPI, since input resolution is the dominant cost lever for vision workloads.

Why do LLM benchmark numbers get contaminated by caching?▼

Inference servers cache aggressively, so reused prompts produce inflated throughput that can be wrong by a factor of 3 or more. The harness restarts the container for a cold cache, gives every request a unique payload, and validates that the prefix cache hit rate stays near 0% before any number is reported.

What happens if a benchmark session is interrupted before teardown?▼

State files are written before stack creation, so teardown can resume after a lost session, and a SessionStart hook warns about instances left running. A teardown sweep lists orphaned stacks you own for confirmation-based deletion, and an on-instance watchdog self-terminates after a configurable lifetime as a backstop.

Can I run this benchmark against a production AWS account?▼

No, it is explicitly designed for sandbox, development, or test accounts only. GPU instances cost roughly $2-15/hr and sometimes over $100/hr, and the load generator saturates its target endpoint by design, so it must never point at shared or production infrastructure.

Related Skills

Explore More Software Engineering Skills→
testdriveraitestdriverai
238

testdriver:aws-setup

Deploy AWS infrastructure with CloudFormation templates for TestDriver test runners.

Official
Advanced
rawwerksrawwerks

openrouter

Route and compare model outputs across OpenRouter's unified API.

Community
Intermediate
OpenDCAIOpenDCAI
114

bench

Benchmark LLM API latency and throughput across streaming endpoints.

Official
Advanced
mahg-esmahg-es
3

llm-local-deploy

Deploy local LLMs with llama.cpp, Ollama, or vLLM via docker-compose.

Official
Intermediate