genie-benchmark-evaluator

Evaluate Genie Space SQL generation accuracy by running benchmark queries against the space and using MLflow GenAI evaluation to log metrics. This process employs a 3-layer judge architecture with 8 scorers to assess quality and correctness.

Measuring SQL generation regressions in a Genie Space requires quantitative benchmark evaluation before and after changes. The evaluation logs results to MLflow, allowing you to compare metrics and identify any accuracy drops prior to deployment.

Yes, you can automate benchmark correction during Genie Space evaluation. The evaluation architecture includes an arbiter layer that can automatically update benchmarks or suggest metadata optimizations based on the correctness assessment outcomes.

Genie Space evaluation works directly with MLflow to log all evaluation metrics, results, and traces. This integration provides detailed analysis and version tracking for your SQL generation accuracy assessments within the MLflow environment.

To run an evaluation on your Genie Space, you need a predefined set of benchmark queries in a YAML file and your Genie Space ID. The evaluation supports both job-based and inline execution modes to process these benchmarks.