skills.rest
SharpAI
SharpAIOFFICIAL
Explore all skills by this author →
GitHub

Cloud Provider Regression Test

Tests cloud LLM providers for connectivity, chat, JSON output, and SSE streaming.

3.0k|478|Updated Mar 5, 2019
Search Tags:#regression testing#API Testing#openai-compatible#lLM#cloud providers#sse streaming#Connectivity
One-click install
npx skills add https://github.com/SharpAI/DeepCamera --skill cloud-provider-regression-test
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: Cloud Provider Regression Test
Source: https://github.com/SharpAI/DeepCamera/tree/main/skills/analysis/cloud-provider-regression
Command: npx skills add https://github.com/SharpAI/DeepCamera --skill cloud-provider-regression-test

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

Verifying that every configured cloud LLM provider still works after config changes, key rotations, or API updates is tedious manual work. This Skill automates regression testing across all enabled providers so broken credentials, wrong base URLs, or incompatible parameters are caught immediately.

Core Features & Use Cases

  • Multi-Provider Testing: Runs connectivity, chat completion, JSON output, and SSE streaming tests against every enabled provider in ~/.aegis-ai/llm-config.json, including OpenAI-compatible, Anthropic, and Gemini API formats.
  • Resilient Request Handling: Automatically retries with legacy max_tokens parameters, backs off on 429 rate limits, and normalizes base URLs with version paths.
  • Structured Reporting: Emits JSONL protocol events for the Aegis runtime and saves timestamped JSON result files to ~/.aegis-ai/regression-tests/.
  • Use Case: After adding a new GLM API key, run the regression suite with a provider filter to confirm chat, JSON, and streaming all pass before using it in production camera alerts.

Quick Start

Run the cloud provider regression test for all enabled providers and show me which ones pass or fail.

Frequently Asked Questions about Cloud Provider Regression Test

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I test cloud LLM provider connectivity and streaming?▼

Run node scripts/run-regression.cjs to test every enabled provider for chat completion, JSON output, and SSE streaming. Use --provider glm,xai to test specific providers or --verbose to see full API responses.

How do I test only specific LLM providers?▼

Pass a comma-separated list with the --provider flag, for example --provider glm,xai, or set the providers parameter in the skill config. Only enabled providers with API keys in llm-config.json are tested.

Which LLM API formats does the regression test support?▼

It supports OpenAI-compatible chat completions, the Anthropic Messages API, and the Gemini generateContent API. Streaming tests run only for OpenAI-compatible providers; Anthropic and Gemini skip the stream test.

Where does the regression test read API keys from?▼

It reads provider configurations and API keys from ~/.aegis-ai/llm-config.json. Providers without an enabled flag or API key are skipped, and results are saved to ~/.aegis-ai/regression-tests/.

What happens when a provider rate limits the test requests?▼

On HTTP 429 responses the script retries up to three times with exponential backoff starting at two seconds. It also retries with the legacy max_tokens parameter if max_completion_tokens is rejected with a 400 error.

Related Skills

Explore More Software Engineering Skills→
syeda-hoorain-alisyeda-hoorain-ali

llm-provider-configuration

Configure the OpenAI Agents SDK with a custom base URL and API key for alternative LLM providers.

Community
Intermediate
RDCrystalLabRDCrystalLab

debug-hypothesis

Structure debugging investigations with evidence-driven root cause analysis.

Official
Advanced
AGMO-IncAGMO-Inc
4

benchmark

Execute benchmark commands and store latency and throughput metrics in Obsidian vaults.

Official
Advanced
diagnostikondiagnostikon
34

opengork

Operate an OpenGork agent with local Ollama or xAI API backends.

Community
Advanced