Generate characterization tests and Promptfoo eval suites for CI ship-gates.
npx skills add https://github.com/Houseofmvps/ultraship --skill evals-houseofmvps
Please help me install this Agent Skill. Skill: evals Source: https://github.com/Houseofmvps/ultraship/tree/main/skills/evals Command: npx skills add https://github.com/Houseofmvps/ultraship --skill evals-houseofmvps
AI-written code often passes code review but fails silently at runtime, and AI-powered features like chatbots, RAG systems, and classifiers drift in behavior as prompts or models change, leading to unexpected production issues that are hard to detect early.
Use the evals skill to build a complete regression and eval test suite for the AI customer support chatbot in the ./src/chat directory before you update its underlying prompt logic.
High-intent search queries and answers about installing and using this skill.
To prevent silent failures when refactoring AI-generated code, generate characterization tests to lock current observable behavior. This immediately catches regressions in AI-written code before deployment, integrating with common runners like Vitest, Jest, and pytest.
Test LLM features for prompt drift by building Promptfoo eval suites that assert chatbots stay on-topic, avoid PII leaks, match formats, and meet latency thresholds. Wire these evals into CI pipelines to block failing changes from deployment.
The best way to lock RAG system behavior is generating Promptfoo eval suites to assert feature behavior. This creates regression tests ensuring your AI features remain stable and compliant before you update underlying prompt logic.
Yes, you can wire promptfoo evals and characterization test runs directly into your CI pipelines as mandatory ship-gates. This integration blocks deployments when tests fail, turning the regression suite into a required code review step.
Yes, characterization testing supports all common test runners including Vitest, Jest, and pytest. It locks the current observable behavior of your codebase before refactoring, catching regressions immediately within your established testing environment.
quanganh208Automate unit, integration, e2e, and UI tests across multiple runners.
bmsull560Automate production release gates with a 7-phase pytest-driven TDD process.
daymadeConfigure and run Promptfoo LLM evaluation workflows with YAML configs and CLI commands.
AngelP17Automate CI/CD pipelines, infrastructure provisioning, and deployments across AWS, GCP, and Azure.
