skill-test

Name: skill-test
Availability: InStock
Author: databricks-solutions

Create, run, and review Databricks skill tests via a YAML-first GRP pipeline.

1.8k|404|Updated Dec 17, 2025

Search Tags:#testing #yaml #MLflow #Databricks #skill-test #GRP

Ready to run

npx skills add https://github.com/databricks-solutions/ai-dev-kit --skill skill-test

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: skill-test
Source: https://github.com/databricks-solutions/ai-dev-kit/tree/main/.test
Command: npx skills add https://github.com/databricks-solutions/ai-dev-kit --skill skill-test

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires python-dotenv, PyYAML, mlflow, databricks-sdk, and includes scripts (resource) and references (resource) components.

What problem does it solve?

The Skill Testing Framework centralizes Databricks skill evaluation by providing a reproducible, YAML-first workflow for generating test scaffolding, executing tests (including optional Databricks MCP), and promoting results to ground truth for reliable regression testing.

Core Features & Use Cases

YAML-driven test definitions and a Generate-Review-Promote (GRP) pipeline to streamline skill validation.
Interactive CLI for creating test prompts, running skills, and reviewing candidates with MLflow-backed metrics.
Ground-truth baselines and MLflow integration to track regressions across skill versions and environments.
Routing and trace evaluation support to validate skill activation and Claude Code behavior.

Quick Start

Install and configure the skill-test CLI in your project (pip install -e ".test/").
Initialize testing scaffolding for a skill with /skill-test <skill-name> init, then run evaluations with /skill-test <skill-name> run.
Use ground_truth.yaml and baseline.yaml to manage regression tests and track improvements over time.

Frequently Asked Questions about skill-test

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I run regression tests for Databricks skills using YAML?▼

Run regression tests for Databricks skills by defining test cases in YAML files and executing the Generate-Review-Promote pipeline to validate behavior and promote results to ground truth baselines.

How does MLflow integration work with skill testing frameworks?▼

MLflow integration tracks evaluation metrics and trace analysis during skill testing, allowing you to monitor regressions across skill versions and environments using baseline comparisons.

Can I use Claude Code routing and trace evaluation to validate skill activation?▼

Yes, the skill testing framework supports routing and trace evaluation to validate skill activation and verify Claude Code behavior through MLflow-backed metrics and interactive CLI reviews.

What is the Generate-Review-Promote pipeline for YAML test definitions?▼

The Generate-Review-Promote pipeline is a YAML-first workflow that generates test scaffolding, executes skill tests, and promotes reviewed candidates to ground truth for reliable regression testing.

Do I need the Databricks SDK and python-dotenv to run skill tests?▼

Yes, the framework requires python-dotenv, PyYAML, mlflow, and databricks-sdk dependencies to execute tests, manage YAML configurations, and integrate with Databricks environments.

How do I initialize and execute a skill test from the CLI?▼

Initialize test scaffolding with `/skill-test <skill-name> init`, then execute evaluations using `/skill-test <skill-name> run` to generate prompts and review candidates against baselines.