skill-test

Evaluate and optimize Databricks agent skills against ground truth YAML test cases.

1|Updated Apr 24, 2026

Search Tags:#optimization #evaluation #yaml #mlflow #databricks #trace analysis #gepa

Ready to run

npx skills add https://github.com/itsadijmbt/SecureMCP-Servers --skill skill-test-itsadijmbt

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: skill-test
Source: https://github.com/itsadijmbt/SecureMCP-Servers/tree/main/TEST_SERVERS/PORTED_TO_SECUREMCP/databrickslab-mcp/ai-dev-kit/.test
Command: npx skills add https://github.com/itsadijmbt/SecureMCP-Servers --skill skill-test-itsadijmbt

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pyyaml, mlflow[databricks]>=3.10.1, protobuf>=5.26, python-dotenv>=1.0.0, gepa>=0.1.0, tiktoken>=0.7.0, litellm<=1.82.6, claude-agent-sdk>=0.1.39, databricks-sdk>=0.20.0, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill solves the challenge of measuring whether Databricks-focused agent instructions (SKILL.md) actually improve real outputs versus running without them, while providing repeatable ground-truth evaluation and iterative optimization.

Core Features & Use Cases

Ground-truth driven evaluation: Runs deterministic assertions and LLM-based judges against ground_truth.yaml to produce pass/fail quality signals.
GEPA-based optimization: Uses evolutionary optimization to mutate SKILL.md based on judge rationales and assertion failures.
Regression testing and baselines: Saves baseline evaluation runs and compares future results to detect regressions.
Trace evaluation support: Validates agent tool usage, token budgets, and execution behavior using Claude Code traces from MLflow or local JSONL.
Workflow automation for test cases: Supports init, add, interactive review, and generate-review-promote flows to maintain high-quality test sets.

Quick Start

Run: /skill-test databricks-spark-declarative-pipelines run

Frequently Asked Questions about skill-test

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I evaluate Databricks agent instructions against ground truth test cases?▼

You can evaluate Databricks agent instructions by running deterministic assertions and LLM-based judges against a ground_truth.yaml file to produce pass/fail quality signals for WITH-skill vs WITHOUT-skill responses.

How does GEPA optimization improve agent instruction skills?▼

GEPA optimization uses an evolutionary loop to mutate SKILL.md files based on judge rationales and assertion failures, iteratively improving Databricks agent behaviors through automated instruction refinement.

Can I use MLflow to score and trace Databricks agent evaluations?▼

MLflow-compatible scoring is fully supported for evaluation runs, and you can validate agent tool usage and token budgets using Claude Code traces from MLflow or local JSONL files.

What is the best way to detect regressions in Databricks agent skills?▼

Regression testing for Databricks agent skills works by saving baseline evaluation runs and comparing future results against them to detect quality degradations across instruction iterations.

Does this YAML-first evaluation approach support interactive test case management?▼

The workflow supports interactive review and generate-review-promote flows to build and maintain high-quality ground_truth.yaml and manifest.yaml datasets for Databricks-focused agent behaviors.

What limitations exist when optimizing Databricks skills with trace-based evaluation?▼

Trace-based evaluation requires Claude Code transcripts from either MLflow or local JSONL, and optimization depends on structured judge rationales from ground_truth.yaml datasets to drive GEPA mutation loops effectively.