Tool in benchmark · Version 1

BlazeSQL in AI Database Agents

Scenario-level performance from current published Results.

21 scenarios with published Results · 28 scenarios in the benchmark

How BlazeSQL performed

Open a capability to explore its scenarios. Each row reports the test set in its published Result; counts are not combined into an overall score.

Analytics & Observability4 scenarios · 2 with published Results
ScenarioPublished outcomesTest coverageResult
Reported analytics match what actually happened
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

BlazeSQL did not report analytics that matched what happened. Its query review page listed only one of five questions and left out both deliberately unanswerable ones, with no answered-versus-unanswered breakdown.

1/1 assessed1/1 gradablePublished test setView Result →
The operator needs to find the questions the agent is failing on
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

BlazeSQL failed to surface unanswered questions as failures. In the recording, Query review showed one ordinary row marked 'Not verified' and no failures or unanswered view appeared anywhere in the sections shown. The social-media complaints question appeared in Chat with a normal answer, and the other unanswerable question is not shown in the admin view.

1/1 assessed1/1 gradablePublished test setView Result →
A user reports a bad answer and the operator has to find itNo published result
Someone needs to see what people have been askingNo published result
Answer Presentation4 scenarios · 3 with published Results
ScenarioPublished outcomesTest coverageResult
The answer is a plain fact or a short list
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL answered the plain fact in prose, so the evidence does not show the result being confined to a chart or a one-cell table. In the captured run, the reply said there were 100 orders ever, and the same frame also showed a one-row result table and no chart.

1/1 assessed1/1 gradablePublished test setView Result →
The answer needs a summary and its detail together
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

BlazeSQL failed to give the overall figure together with the breakdown. It first asked which dimension to use, then showed category-level query, results, highlights, and CSV export, but none stated the month total.

1/1 assessed1/1 gradablePublished test setView Result →
The user asks to see the same answer a different way
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL passed this scenario: it showed the same revenue answer in a different form. In the recorded conversation, the first result appeared as a table with seven categories, then the follow-up request switched that same result to a bar chart. The bar lengths matched the table values within the chart's scale.

1/1 assessed1/1 gradablePublished test setView Result →
The answer is a trend or a comparisonNo published result
Conversational Interaction6 scenarios · 5 with published Results
ScenarioPublished outcomesTest coverageResult
Customer corrects information given earlier
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL replaced the earlier July answer with the corrected June answer. In the visible exchange, the first turn returned 21 orders for July 2026, and the follow-up turn rebuilt the query for June 2026 and returned 20 orders, matching the corrected period only.

1/1 assessed1/1 gradablePublished test setView Result →
The agent offers what to ask next
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL offered what to ask next after answering the revenue question. It ended the August 2026 revenue reply with “Would you like a breakdown by category, or any other analysis?”, so the capture shows one specific follow-up rather than only a generic closing.

1/1 assessed1/1 gradablePublished test setView Result →
The follow-up is ambiguous
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL asked which comparison the customer meant instead of silently choosing one. Turn 1 showed two distinct candidate sets, and turn 2 returned only a clarification question with no table, chart, or figures. It also produced a merged comparison view in turn 1.

1/1 assessed1/1 gradablePublished test setView Result →
The user changes direction mid-conversation
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

BlazeSQL failed to shift from the first-turn Texas filter to the broader all-states answer. In the second turn it dropped the filter but returned a state-by-state breakdown instead of the overall 12% rate; the export totals 100 orders and 12 cancellations, so the missing figure was available.

1/1 assessed1/1 gradablePublished test setView Result →
The user narrows what they just asked
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL preserved the earlier 90-day window when the user narrowed the request to Texas. The follow-up query kept the original date filter and added only the Texas filter, and the Texas export matched the Texas subset of the first export.

1/1 assessed1/1 gradablePublished test setView Result →
The next question refers to the previous answerNo published result
Data Access Control3 scenarios · 2 with published Results
ScenarioPublished outcomesTest coverageResult
An excluded table is needed to answer the question
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

BlazeSQL fails when the needed table is excluded. It never said it could not answer; instead it returned 4 from support tickets and then widened the search to pending confirmations, action log, and orders, while still never querying the excluded table.

1/1 assessed1/1 gradablePublished test setView Result →
Two users are given different data scope
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL kept the two scopes apart. The full-access account returned plus for the membership question, while the restricted account refused to look the customer up and showed no SQL or result table.

1/1 assessed1/1 gradablePublished test setView Result →
An excluded column is asked for directlyNo published result
Question Answering4 scenarios · 3 with published Results
ScenarioPublished outcomesTest coverageResult
The answer is available in the connected dataDifferent test setPublished Result available, but not included in this comparison.View Result →
The connected data cannot answer the question
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL said it could not answer from the available data when asked how many 5-star reviews were received this month. It listed ten available tables, gave no number, and pointed to adding review data or using a related source instead. The key finding is that it declined rather than inventing a count.

1/1 assessed1/1 gradablePublished test setView Result →
The question uses a term the company defines itself
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL did not answer revenue as a bare number; it stated the definition it applied and showed the breakdown it used. In the captured Sep 2, 2026 run, it reported order revenue, subscription revenue, refunds, and an adjusted net revenue total, with the order line explicitly marked as excluding cancelled orders.

1/1 assessed1/1 gradablePublished test setView Result →
The user's wording doesn't match how the values are storedNo published result
Reporting3 scenarios · 3 with published Results
ScenarioPublished outcomesTest coverageResult
The report is run again after the data has changed
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL kept the saved report current after new rows were added. The fixed-date save rose from 52 to 55, and the relative 30-day save rose from 162 to 164.

1/1 assessed1/1 gradablePublished test setView Result →
The user wants several findings in one place
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL passed this one-report task. The combined table kept the earlier revenue, category, and cancellation answers intact, with the same values and bases, and brought all five metrics into one report.

1/1 assessed1/1 gradablePublished test setView Result →
The user wants to keep an answer they just got
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL saves an answer and its query, then reopens the saved item with the same three rows under the saved name. The saved item appears in a separate Saved area, and the reopened table matches the earlier chat answer.

1/1 assessed1/1 gradablePublished test setView Result →
Training4 scenarios · 3 with published Results
ScenarioPublished outcomesTest coverageResult
The database's names are cryptic and the user documents them
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL used the user’s documentation to answer from the cryptic table and column. It reported 6 returns currently awaiting manual fraud review, split across 5 approved and 1 received. The stored note was applied to returns.flg_x2, and the follow-up answer came from that documented column.

1/1 assessed1/1 gradablePublished test setView Result →
The user sets a standing rule for all future answers
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL kept the standing rule across future questions. After the rule was set once, the later revenue answer returned 51,848.02 and the later order-count answer returned 32, and both generated queries still filtered out cancelled orders.

1/1 assessed1/1 gradablePublished test setView Result →
The user supplies example questions and the queries they trust
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL generalized the supplied question-query pair in this case. It first asked what counted as successful, then after the example query was added it returned 42 for both the repeated question and a reworded one; the same-session reference value was 42.

1/1 assessed1/1 gradablePublished test setView Result →
The agent used the wrong definition and the user corrects itNo published result

Reading these Results

Published evidence and test coverage answer different questions.

Publication availability

Which scenarios have a Result?

A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.

Test coverage

What does each Result cover?

Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.

Scenario scope

Inventory is not testing progress

The 28 scenarios describe this benchmark’s scope. They are not an assumed applicability or test-coverage denominator for BlazeSQL.

BlazeSQL in AI Database Agents | AI Demos