skills.rest
Kiln-AI
Kiln-AIOFFICIAL
Explore all skills by this author →
WebsiteGitHub

kiln-prerelease-check

Runs Kiln prerelease smoke tests and CI checks, then diagnoses failures into a report.

5.0k|376|Updated Jul 23, 2024
Search Tags:#smoke tests#release validation#PyTest#prerelease#ci checks#LLM providers#model deprecation
One-click install
npx skills add https://github.com/Kiln-AI/Kiln --skill kiln-prerelease-check
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: kiln-prerelease-check
Source: https://github.com/Kiln-AI/Kiln/tree/main/.agents/skills/kiln-prerelease-check
Command: npx skills add https://github.com/Kiln-AI/Kiln --skill kiln-prerelease-check

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Before tagging a Kiln release, someone must run the paid live-API smoke tests, figure out why each failure broke (stale model pin vs. real regression vs. transient flake), and check whether hardcoded model slugs have gone stale — a slow, error-prone manual audit.

Core Features & Use Cases

  • Full prerelease verification: Runs checks.sh (lint/format/typecheck/unit tests) plus the curated @pytest.mark.prerelease live-API test suite across OpenAI, Anthropic, Gemini, Vertex, Groq, OpenRouter, Bedrock, Fireworks, and more.
  • Failure diagnosis with retry: Classifies each failure as deprecated model, provider-removed slug, provider behavior change, Kiln code regression, or auth/transient, retrying likely flakes once and disclosing every retry.
  • Model-pin staleness sweep: Cross-checks every whitelisted and hardcoded model slug against ml_model_list.py for deprecations and newer siblings, plus prod-code probe slugs.
  • Read-only reporting: Writes a single verdict-first REPORT.md under .prerelease/<timestamp>/ with recommended actions; never edits code, tests, or whitelists.
  • Use Case: Before tagging a Kiln release, ask for a prerelease check to get one report covering CI status, live-API test results, stale model pins, coverage gaps, and an ordered list of pre-release actions.

Quick Start

Run the Kiln prerelease check on this repository and give me the verdict report before I tag the release.

Frequently Asked Questions about kiln-prerelease-check

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run prerelease smoke tests before a Kiln release?▼

Invoke the skill and it runs checks.sh plus `pytest --runprerelease`, which executes the curated set of paid live-API tests serially. Results, diagnoses, and recommendations are written to `.prerelease/<timestamp>/REPORT.md`.

How do I tell a stale model pin from a real regression in live API tests?▼

The skill classifies each failure: provider 'model not found' errors are cross-referenced against ml_model_list.py deprecation status, while changed response shapes or tracebacks in Kiln code are flagged as regressions. Ambiguous cases are reported as possible regressions with a disambiguation step.

Does the prerelease check modify any code or test files?▼

No, it is strictly read-only. It writes only to the gitignored `.prerelease/<timestamp>/` directory, and every fix — pin bumps, whitelist updates, deprecation flags — appears as a recommendation in the report for the user to apply.

What API keys are needed for the prerelease test suite?▼

Keys for OpenAI, Anthropic, Gemini, Groq, OpenRouter, Fireworks, Together, Cohere, and AWS may be used, sourced from a `.env` file. Missing keys cause tests to skip rather than fail, and skips are listed by provider in the report.

Why did a live API test fail once but pass on retry?▼

Live-API tests are prone to transient flakes like rate limits, timeouts, and dropped streams. The skill retries likely-transient failures once in isolation; a pass on retry is reported as transient/flaky with both runs disclosed, never silently swapped.

Related Skills

Explore More Software Engineering Skills→
rhpdsrhpds

01-vertex-setup

Configure Claude Code to use Google Cloud Vertex AI as the backend provider.

Official
Advanced
ShareXShareX
8

publish-release

Automate the ShareX.ImageEditor release workflow with version bump, tagging, and monitoring.

Official
Advanced
Tr3kkRTr3kkR
15

yuzu-build

Automate Yuzu setup, build, and testing across Linux, macOS, and Windows.

Community
Intermediate
googlegoogle
68

dev-capsem-doctor

Run pytest-based diagnostic suites inside Capsem VMs to validate sandbox health.

Official
Advanced