Model Manager

Test API access and update models.yml for new AI models.

33|5|Updated Sep 26, 2025

Search Tags:#automation #testing #evaluation #configuration #api-testing #ai-models

Ready to run

npx skills add https://github.com/sunholo-data/ailang --skill model-manager

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: Model Manager
Source: https://github.com/sunholo-data/ailang/tree/main/.claude/skills/model-manager
Command: npx skills add https://github.com/sunholo-data/ailang --skill model-manager

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires curl, python3, jq, gcloud, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill eliminates the complex, error-prone manual process of testing and adding new AI models to evaluation suites, reducing setup time by 70%.

Core Features & Use Cases

API Access Testing: Verify authentication and connectivity for OpenAI, Anthropic, and Google models.
Configuration Management: Safely update models.yml with automated validation and backup creation.
Use Case: When your team needs to add GPT-5.1 to the eval suite, this Skill automatically tests access, finds pricing, updates configs, and runs verification benchmarks.

Quick Start

Use the Model Manager skill to test API access for the new GPT-5.1 model.

Frequently Asked Questions about Model Manager

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I automate testing and deployment of new AI models across multiple providers?▼

Model Manager automates AI model testing by verifying API access, retrieving pricing, and validating configuration across OpenAI, Anthropic, and Google Vertex AI. It reduces setup time by 70% through scripted validation, models.yml updates, and end-to-end readiness verification without manual error-prone steps.

Can I validate model availability and pricing for OpenAI, Anthropic, and Google models automatically?▼

Yes. Model Manager tests API connectivity for all three providers, extracts current pricing and model metadata, and updates your models.yml with correct friendly names and provider details through automated scripts, eliminating manual lookup and configuration errors.

What's the best way to add a new model to an evaluation suite?▼

Use Model Manager to run test_model_access.sh for API validation, find_model_info.sh for pricing discovery, update_models_yml.sh for safe configuration updates with backups, and run_test_benchmark.sh to verify integration. The Skill handles all steps end-to-end with environment prerequisite checks.

Do I need gcloud, Python, and curl to use model testing automation?▼

Yes. Model Manager requires curl for API calls, Python3 for script execution, jq for YAML parsing, and gcloud for Google Vertex AI authentication. These dependencies enable testing across all three provider platforms and configuration management.

How does automated model configuration validation prevent deployment errors?▼

Model Manager validates models.yml integrity before updates, creates automatic backups, verifies API access before integration, and runs benchmark tests post-deployment. This scripted validation catches misconfigurations, missing credentials, and pricing mismatches before they affect your evaluation suite.