skills.rest
databricks-solutions
databricks-solutionsOFFICIAL
Explore all skills by this author →
GitHub

genie-benchmark-generator

Generate and validate benchmark questions with SQL execution and MLflow synchronization.

5|6|Updated Jan 26, 2026
Search Tags:#optimization#benchmark#mlflow#genie#sql validation#genai evaluation
One-click install
npx skills add https://github.com/databricks-solutions/vibe-coding-workshop-template --skill genie-benchmark-generator
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: genie-benchmark-generator
Source: https://github.com/databricks-solutions/vibe-coding-workshop-template/tree/main/data_product_accelerator/skills/semantic-layer/genie-optimization-workers/01-genie-benchmark-generator
Command: npx skills add https://github.com/databricks-solutions/vibe-coding-workshop-template --skill genie-benchmark-generator

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill automates the creation, validation, and synchronization of benchmark questions for Genie Space optimization, ensuring robust evaluation of AI agent performance.

Core Features & Use Cases

  • Multi-path Intake: Handles user-provided questions (10+, 1-9), or generates synthetic benchmarks from scratch.
  • Ground Truth Validation: Executes SQL queries against the live warehouse to verify correctness and store results.
  • MLflow Integration: Syncs validated benchmarks to MLflow Evaluation Datasets for seamless integration with GenAI evaluation workflows.
  • Use Case: Before optimizing a Genie Space for cost analysis, use this Skill to generate a comprehensive set of benchmark questions, validate their expected SQL, and prepare them for MLflow evaluation, ensuring the optimization loop has reliable test cases.

Quick Start

Use the genie-benchmark-generator skill to create and validate benchmarks for the 'cost' domain in the 'main.genie_benchmarks' Unity Catalog schema.

Frequently Asked Questions about genie-benchmark-generator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I generate and validate benchmark questions for Genie Space optimization?▼

Generate benchmarks from user-provided or synthetic questions, validate ground truth SQL via warehouse execution, store result hashes, and synchronize validated test cases to MLflow Evaluation Datasets in Unity Catalog.

How does SQL validation work when creating GenAI evaluation datasets?▼

Ground truth SQL validation executes queries against a live warehouse to verify correctness, stores result hashes, and ensures benchmark reliability for GenAI evaluation loops.

Can I use my own questions to build MLflow evaluation datasets in Unity Catalog?▼

Yes, the multi-path intake handles user-provided questions (10 or more, or 1 to 9), validates ground truth SQL via warehouse execution, and synchronizes them to MLflow Evaluation Datasets in Unity Catalog.

What is the best way to prepare test cases for Genie Space AI agent evaluation?▼

Generate comprehensive benchmark questions, validate expected SQL outputs via warehouse execution, and sync results to MLflow Evaluation Datasets to ensure the Genie Space optimization loop has reliable GenAI evaluation test cases.

Do I need a live warehouse to validate ground truth SQL for Genie benchmarks?▼

Yes, a live warehouse is required because ground truth SQL validation executes queries against it to verify correctness and store result hashes before synchronizing to MLflow Evaluation Datasets in Unity Catalog.

Related Skills

Explore More Data & Analytics Skills→
thbehthbeh

databricks-genie

Create and query Databricks Genie Spaces for natural language SQL exploration.

Community
Advanced
prashsubprashsub
1

genie-benchmark-generator

Generate and validate Genie Space benchmark questions with ground truth SQL.

Community
Advanced
mirakuimirakui

databricks-mlflow-evaluation

Configure Databricks MLflow GenAI evaluation workflows with mlflow.genai.evaluate().

Community
Advanced
flashinfer-aiflashinfer-ai
264

add-reference-tests

Generate and validate pytest tests for flashinfer_trace reference implementations against FlashInfer or SGLang ground truth.

Official
Advanced