How to Evaluate AI-Generated SQL on Databricks

An AI assistant can produce a query that runs and returns a plausible result.

But how do we know if it's correct?

Suppose a user asks:

Which enterprise customers had revenue growth last quarter?

An AI assistant might generate something like this:

SELECT
  c.customer_id,
  c.customer_name,
  SUM(o.revenue) AS revenue
FROM main.sales.orders o
JOIN main.crm.customers c
  ON o.customer_id = c.customer_id
WHERE c.segment = 'enterprise'
  AND o.order_date >= date_trunc('quarter', current_date()) - INTERVAL 3 MONTH
GROUP BY c.customer_id, c.customer_name
ORDER BY revenue DESC;

At first glance, this looks reasonable. It joins customers to orders, filters for enterprise accounts, and looks at last quarter.

But it does not answer the question.

The user asked for customers with revenue growth. This query only calculates revenue for one period. It never compares last quarter to the previous quarter. It could return the highest-revenue customers even if their revenue declined.

It's a subtle mistake, and one that would be difficult to catch without an evaluation framework.

What A SQL Eval Should Catch

Good SQL evals should catch:

• Bad assumptions: wrong tables, hallucinated columns, or stale schema context.

• Business logic errors: revenue vs bookings, gross vs net, churn date vs cancellation date.

• Filter and time-window errors: missing segment filters, current quarter vs last completed quarter, inclusive/exclusive date bounds.

• Join and aggregation errors: duplicated rows, wrong grain, or aggregating at the wrong point in the query.

In the example above, a better query would compare two periods:

WITH quarterly_revenue AS (
  SELECT
    c.customer_id,
    c.customer_name,
    SUM(CASE
      WHEN o.order_date >= add_months(date_trunc('quarter', current_date()), -3)
       AND o.order_date < date_trunc('quarter', current_date())
      THEN o.revenue ELSE 0
    END) AS last_quarter_revenue,
    SUM(CASE
      WHEN o.order_date >= add_months(date_trunc('quarter', current_date()), -6)
       AND o.order_date < add_months(date_trunc('quarter', current_date()), -3)
      THEN o.revenue ELSE 0
    END) AS previous_quarter_revenue
  FROM main.sales.orders o
  JOIN main.crm.customers c
    ON o.customer_id = c.customer_id
  WHERE c.segment = 'enterprise'
  GROUP BY c.customer_id, c.customer_name
)

SELECT
  customer_id,
  customer_name,
  last_quarter_revenue,
  previous_quarter_revenue,
  last_quarter_revenue - previous_quarter_revenue AS revenue_growth
FROM quarterly_revenue
WHERE last_quarter_revenue > previous_quarter_revenue
ORDER BY revenue_growth DESC;

This version may still need refinement depending on the business definition of "revenue," "enterprise customer," and "last quarter." But it at least expresses the comparison the user asked for.

SQL evals check whether generated SQL matches the intended behavior, not only whether it compiles.

The Unit Of A SQL Eval

A SQL eval usually needs four things:

1. A natural-language question.

2. The available schema or database context.

3. A generated SQL query.

4. An expected behavior.

The expected behavior can be represented in different ways.

Sometimes you know the exact SQL query you want. Sometimes you only care about the result set. Sometimes several queries are acceptable as long as they are semantically equivalent. Sometimes the right test is not "does this match exactly?" but "does this query correctly apply the business rule?"

For example:

Question:
Which enterprise customers had revenue growth last quarter?

Expected behavior:
Return enterprise customers whose revenue in the most recently completed quarter
was greater than their revenue in the quarter before that.

Required checks:
- Filters customer segment to enterprise.
- Compares last completed quarter against previous quarter.
- Aggregates revenue at the customer level.
- Excludes customers with flat or declining revenue.

A golden SQL query can be a useful reference, especially when it captures the intended logic clearly. But exact string matching is usually too brittle on its own.

One model might use CTEs. Another might use window functions. Another might use date dimension tables. Depending on the case, you might want to compare query structure, execute both queries and compare the results, check output properties, or use an LLM judge for more ambiguous cases.

A good SQL eval framework should support those different methods and let you choose the one that fits the question.

How To Run SQL Evals On Databricks

A practical SQL eval starts with a small set of real questions your users ask.

For each question, define what a correct answer means. That might be an expected result set, a known-good SQL query, or a set of behavioral checks.

For example:

Question	Expected behavior	Checks
Which enterprise customers had revenue growth last quarter?	Compare the most recently completed quarter against the quarter before it	enterprise filter, customer-level aggregation, positive growth only
Which customers churned in the last 30 days?	Return customers whose churn date falls in the last 30 days	use churn date, exclude test accounts, apply correct date window
What were our top 10 products by gross margin last month?	Rank products by revenue minus COGS for the last completed month	use gross margin, group by product, sort descending, limit to 10

The exact format depends on your eval framework. The important part is making the expected behavior explicit enough that you can re-run the test when your prompt, model, schema, or business logic changes.

A typical workflow looks like this:

1. Collect representative natural-language questions.

2. Define the expected SQL behavior for each question.

3. Generate SQL with your assistant or model.

4. Run the SQL against Databricks.

5. Compare the result to the expected behavior.

6. Track failures and regressions over time.

A Databricks Eval In evaldata

evaldata is a pytest-native framework for evaluating AI-generated SQL. It can run SQL in Databricks, compare result sets, assert expected schema, and push structural checks down into the warehouse.

Here is the shape of a Databricks eval from the bundled Databricks example:

import os
from decimal import Decimal

from evaldata import (
    CallableSolver,
    EvalCase,
    ExpectationSuiteScorer,
    ResultSetEquivalence,
    assert_eval,
    eval_case,
)
from evaldata.platforms import databricks_platform

platform = databricks_platform(
    name="examples-databricks",
    server_hostname=os.environ["DATABRICKS_SERVER_HOSTNAME"],
    http_path=os.environ["DATABRICKS_HTTP_PATH"],
)


@eval_case(
    input="What is the total order amount?",
    expected={"rows": [{"total": Decimal("35.50")}]},
    platform=platform,
)
def test_total_order_amount(case: EvalCase) -> None:
    solver = CallableSolver(
        lambda c: "SELECT sum(amount) AS total FROM evaldata_ex04_orders"
    )
    assert_eval(case, solver, scorers=[ResultSetEquivalence()])

That example uses fixed SQL so the Databricks behavior is easy to see. In a production eval, the solver can be your SQL assistant, model call, prompt chain, or application code. The eval case stays the same: a user question, an expected answer, a Databricks platform, and a scorer.

The full example also shows schema-aware checks:

@eval_case(
    input="List each order's customer and amount, ordered by id.",
    expected={
        "rows": [
            {"customer": "Ada", "amount": Decimal("10.00")},
            {"customer": "Bo", "amount": Decimal("5.50")},
            {"customer": "Cy", "amount": Decimal("20.00")},
        ],
        "schema": [
            {"name": "customer", "type": "STRING"},
            {"name": "amount", "type": "DECIMAL(10, 2)"},
        ],
    },
    platform=platform,
)
def test_precise_types_resolved(case: EvalCase) -> None:
    solver = CallableSolver(
        lambda c: "SELECT customer, amount FROM evaldata_ex04_orders ORDER BY id"
    )
    assert_eval(case, solver, scorers=[ResultSetEquivalence()])

That matters on Databricks because precise warehouse types can matter. In the example above, evaldata resolves DECIMAL(10, 2) from the warehouse instead of relying only on the driver's generic type description.

The Databricks guide has the full setup: install the Databricks extra, set DATABRICKS_SERVER_HOSTNAME and DATABRICKS_HTTP_PATH, authenticate with the Databricks SDK or CLI, and run the eval with pytest.

uv add "evaldata[databricks]"
uv run pytest examples/04_databricks -q

You can read the full guide in the evaldata docs.

Result Equivalence vs Semantic Equivalence

There are two important kinds of SQL correctness.

The first is result equivalence.

If two queries return the same rows and values on a representative dataset, they may be equivalent for the purposes of the task.

For example, these two queries may produce the same answer:

SELECT customer_id, SUM(revenue)
FROM orders
GROUP BY customer_id;

SELECT customer_id, total_revenue
FROM customer_revenue_summary;

If customer_revenue_summary is trusted and up to date, both might be acceptable.

The second is semantic equivalence.

Semantic equivalence asks whether the query means the same thing as the expected answer, even if the exact syntax and intermediate steps differ.

This matters because AI-generated SQL often has many valid forms. You do not want your eval suite to be brittle. A correct query should pass even if it uses a different join order, alias name, CTE structure, or aggregation style.

But semantic equivalence also needs to be strict enough to catch meaningful mistakes.

These two queries are not equivalent:

WHERE order_date >= date_trunc('quarter', current_date())

WHERE order_date >= add_months(date_trunc('quarter', current_date()), -3)
  AND order_date < date_trunc('quarter', current_date())

The first means "this quarter so far." The second means "last completed quarter." If the question asks for one, the eval should not accept the other.

A good SQL eval should understand that.

evaldata includes semantic equivalence for cases where you can compare SQL structure directly, without executing the query or asking an LLM to judge. When structure alone is not enough, you can execute in the warehouse or use an LLM judge for ambiguous cases.

Data Expectations

Sometimes the expected result is not a fixed set of rows. You may only need to know that the query returns the right shape.

For example, an eval might assert that a result has exactly three rows, that id is never null, and that id is unique. In evaldata, those checks can run against Databricks as an expectation suite:

@eval_case(
    input="List every order's id and customer.",
    expected={
        "kind": "expectation_suite",
        "expectations": [
            {"kind": "row_count", "exact": 3},
            {"kind": "not_null", "column": "id"},
            {"kind": "unique", "column": "id"},
        ],
    },
    platform=platform,
)
def test_expectation_suite_pushdown(case: EvalCase) -> None:
    solver = CallableSolver(
        lambda c: "SELECT id, customer FROM evaldata_ex04_orders"
    )
    assert_eval(case, solver, scorers=[ExpectationSuiteScorer()])

This is from the bundled Databricks example. The row-level checks are pushed into Databricks as SQL, so the warehouse does the work instead of pulling the full result set back into the test process.

Where MLflow Fits

Databricks has strong support for GenAI evaluation through MLflow 3, especially for agents, traces, judges, evaluation datasets, and monitoring. If you are building RAG apps or multi-step agents on Databricks, MLflow is often the natural place to track runs, inspect traces, and monitor quality over time.

AI-generated SQL adds a more specific evaluation problem. The query itself may need checks for equivalence, result-set comparison, schema expectations, warehouse-specific types, join grain, date windows, and business logic encoded in SQL. Those checks can sit alongside broader Databricks and MLflow workflows, especially when prompts, models, schemas, or metric definitions change.

Using evaldata

evaldata is built for that SQL-specific layer. It lets you express common questions as pytest tests, run generated SQL against Databricks, and score the result with the method that fits the case: semantic equivalence, result-set comparison, data expectations, or an LLM judge.

The payoff is regression testing. When you change the prompt, switch models, update schema context, or add a new business definition, you can run the same evals again and see which questions still pass.

Start Small

You do not need a huge benchmark to make SQL evals useful.

Take five real questions your users ask today. Pick questions where a plausible query can still be wrong: revenue growth, churn, active users, margin, retention, or anything with date windows and joins.

For each case, choose the check that fits: semantic equivalence, result-set comparison, data expectations, or an LLM judge. Then run the suite whenever you change the prompt, model, schema context, or metric definition.

The bundled Databricks example in the evaldata repo is a great starting point.