genie-benchmark-generator

To generate Genie Space benchmarks, you can provide user questions or use synthetic generation from trusted assets, validating ground truth SQL against the warehouse to ensure comprehensive query coverage before optimization loops.

Ground truth SQL validation works by executing queries against the warehouse, storing result hashes, and synchronizing the validated data to MLflow Evaluation Datasets to maintain robust optimization cycles.

The best way to validate benchmark questions is by validating ground truth SQL directly against the warehouse and syncing to MLflow Evaluation Datasets, ensuring suites are comprehensive and synchronized for robust testing.

Yes, you can use custom questions for Genie Space evaluation through multi-path intake, which supports both individual user-provided questions and bulk synthetic generation from trusted assets.

Yes, MLflow is needed to synchronize validated benchmark questions into MLflow Evaluation Datasets, ensuring the benchmark suites are properly structured for robust optimization cycles.

You should avoid synthetic generation when you lack trusted assets, as the system relies on these assets to automatically produce diverse and comprehensive benchmark questions for accurate Genie Space evaluation.