sql-eval

Evaluate AI-generated SQL code for correctness, formatting, performance, and reliability.

2|Updated Apr 20, 2026

Search Tags:#code review #performance #analytics #benchmarking #evaluation #sql

Ready to run

npx skills add https://github.com/sanjogkadayat-web/eval-framework --skill sql-eval

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: sql-eval
Source: https://github.com/sanjogkadayat-web/eval-framework/tree/main/sql-eval
Command: npx skills add https://github.com/sanjogkadayat-web/eval-framework --skill sql-eval

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tiktoken, sqlfluff, flake8, pytest, pandas, pyyaml, duckdb, and includes scripts (resource) and assets (resource) components.

What problem does it solve?

This Skill solves the challenge of objectively evaluating AI coding assistants by providing a deterministic, reproducible framework to score SQL query quality across correctness, formatting, performance, and reliability.

Core Features & Use Cases

Quantitative Benchmarking: Scores model-generated SQL against 60 analytics tasks using pre-recorded metrics, ensuring consistent results.
Structured Critic Review: Automatically generates qualitative feedback using a specialized SQL Critic persona to identify logic errors, schema fragility, and style issues.
Use Case: Analytics teams can use this to compare Claude, ChatGPT, and Gemini performance on complex banking SQL tasks before integrating them into production pipelines.

Quick Start

Run the sql-eval skill to evaluate all tasks and generate a comparative scorecard for the three supported models.

Frequently Asked Questions about sql-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I benchmark AI SQL coding assistants for correctness and performance?▼

You can benchmark AI SQL coding assistants by running the sql-eval skill, which scores model-generated SQL queries across correctness, formatting, performance, and reliability using pre-recorded synthetic banking datasets. It evaluates 60 analytics tasks to generate a structured comparative scorecard.

What metrics are used to evaluate AI-generated SQL queries for analytics engineering tasks?▼

Evaluating AI-generated SQL queries involves scoring them across correctness, formatting, performance, and reliability dimensions. The framework specifically assesses complex analytics engineering tasks like CTEs, window functions, and incremental loads using pre-recorded synthetic banking datasets.

Does the SQL evaluation framework support comparing models like Claude, ChatGPT, and Gemini?▼

Yes, the SQL evaluation framework supports comparing AI models like Claude, ChatGPT, and Gemini. It generates a comparative scorecard that allows analytics teams to evaluate performance on complex banking SQL tasks before integrating them into production pipelines.

Can I use duckdb and pandas to process SQL benchmarking evaluation results?▼

Yes, you can use duckdb and pandas to process SQL benchmarking evaluation results. The framework requires pandas, sqlfluff, and duckdb to process the evaluation outputs and generate structured scorecards from the model-generated SQL queries.

What is the best way to automatically review SQL code for logic errors and schema fragility?▼

The best way to automatically review SQL code for logic errors and schema fragility is using a structured critic review mechanism. The framework applies a specialized SQL Critic persona to automatically generate qualitative feedback and identify style issues.

Do I need sqlfluff to evaluate SQL formatting and style in AI benchmarking?▼

Yes, you need sqlfluff to evaluate SQL formatting and style during AI benchmarking. It is a required dependency for processing evaluation results, ensuring that model-generated SQL queries are scored accurately across the formatting dimension.