skills.rest
surus-lat
surus-latOFFICIAL
Explore all skills by this author →
Website𝕏@surusdevGitHub

add-provider

Integrate AI model providers into the Benchy benchmarking framework via YAML configuration.

8|Updated Sep 12, 2025
Search Tags:#testing#llm#benchmarking#configuration#api-integration#ai-models
One-click install
npx skills add https://github.com/surus-lat/benchy --skill add-provider-surus-lat
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: add-provider
Source: https://github.com/surus-lat/benchy/tree/main/.agent/skills/add-provider
Command: npx skills add https://github.com/surus-lat/benchy --skill add-provider-surus-lat

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This skill streamlines the process of extending the Benchy benchmarking engine to support new model providers, whether they are OpenAI-compatible endpoints or custom HTTP-based systems.

Core Features & Use Cases

  • Provider Integration: Easily add new model providers by configuring YAML files for OpenAI-compatible APIs or implementing custom interface classes for proprietary systems.
  • Capability Mapping: Define specific provider capabilities like multimodal support, streaming, or logprobs to ensure accurate benchmarking.
  • Verification: Includes a built-in smoke test workflow to validate new provider configurations before full-scale evaluation.

Quick Start

Follow the instructions in the skill documentation to create the necessary configuration files and interface code for your specific provider type.

Frequently Asked Questions about add-provider

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I add a new AI inference provider to a benchmarking framework?▼

To add a new AI inference provider, you configure provider YAML files for OpenAI-compatible endpoints or implement custom interface classes for proprietary HTTP APIs, defining request-response cycles within the benchmarking framework.

Can I benchmark custom HTTP inference systems that are not OpenAI-compatible?▼

Yes, you can benchmark custom HTTP inference systems by implementing modular custom interface classes that handle the proprietary request-response cycles, rather than relying on standard OpenAI-compatible endpoint configurations.

How do I map specific provider capabilities like streaming or multimodal support for benchmarking?▼

You map specific provider capabilities like multimodal support, streaming, or logprobs by defining them within the provider configuration, ensuring the benchmarking engine accurately tests and evaluates those specific features.

What is the best way to verify a new AI model provider configuration before running full benchmarks?▼

The best way to verify a new AI model provider configuration is by running the built-in smoke test workflow, which validates the setup to ensure proper integration before executing full-scale benchmarking evaluations.

Do I need to write custom interface code to integrate an OpenAI-compatible API endpoint?▼

No, you do not need to write custom interface code for OpenAI-compatible API endpoints; you only need to create and configure the necessary YAML files to integrate the provider into the benchmarking framework.

Related Skills

Explore More Software Engineering Skills→
xianmingyaoxianmingyao
1

http-api-cloudbase

Access CloudBase features via HTTP API without SDKs.

Community
Intermediate
Ankh-StudioAnkh-Studio

batch-evaluation

Process multiple artifacts concurrently to generate comparative analysis and trend reports.

Official
Intermediate
strataljsstrataljs
10

stratal-testing

Compile Stratal modules with provider overrides and simulate HTTP requests.

Official
Intermediate
surus-latsurus-lat
8

evaluate

Executes standardized AI benchmarking workflows with smoke tests and configurable interfaces.

Official
Advanced