BlazeSQL in AI Database Agents
Scenario-level performance from current published Results.
21 scenarios with published Results · 28 scenarios in the benchmark
How BlazeSQL performed
Open a capability to explore its scenarios. Each row reports the test set in its published Result; counts are not combined into an overall score.
Analytics & Observability4 scenarios · 2 with published Results
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Reported analytics match what actually happened | 0 Pass1 Fail0 Not gradable 1 of 1 test case failedWhat happenedBlazeSQL did not report analytics that matched what happened. Its query review page listed only one of five questions and left out both deliberately unanswerable ones, with no answered-versus-unanswered breakdown. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The operator needs to find the questions the agent is failing on | 0 Pass1 Fail0 Not gradable 1 of 1 test case failedWhat happenedBlazeSQL failed to surface unanswered questions as failures. In the recording, Query review showed one ordinary row marked 'Not verified' and no failures or unanswered view appeared anywhere in the sections shown. The social-media complaints question appeared in Chat with a normal answer, and the other unanswerable question is not shown in the admin view. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| A user reports a bad answer and the operator has to find it | No published result | ||
| Someone needs to see what people have been asking | No published result | ||
Answer Presentation4 scenarios · 3 with published Results
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| The answer is a plain fact or a short list | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL answered the plain fact in prose, so the evidence does not show the result being confined to a chart or a one-cell table. In the captured run, the reply said there were 100 orders ever, and the same frame also showed a one-row result table and no chart. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The answer needs a summary and its detail together | 0 Pass1 Fail0 Not gradable 1 of 1 test case failedWhat happenedBlazeSQL failed to give the overall figure together with the breakdown. It first asked which dimension to use, then showed category-level query, results, highlights, and CSV export, but none stated the month total. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The user asks to see the same answer a different way | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL passed this scenario: it showed the same revenue answer in a different form. In the recorded conversation, the first result appeared as a table with seven categories, then the follow-up request switched that same result to a bar chart. The bar lengths matched the table values within the chart's scale. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The answer is a trend or a comparison | No published result | ||
Conversational Interaction6 scenarios · 5 with published Results
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Customer corrects information given earlier | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL replaced the earlier July answer with the corrected June answer. In the visible exchange, the first turn returned 21 orders for July 2026, and the follow-up turn rebuilt the query for June 2026 and returned 20 orders, matching the corrected period only. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The agent offers what to ask next | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL offered what to ask next after answering the revenue question. It ended the August 2026 revenue reply with “Would you like a breakdown by category, or any other analysis?”, so the capture shows one specific follow-up rather than only a generic closing. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The follow-up is ambiguous | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL asked which comparison the customer meant instead of silently choosing one. Turn 1 showed two distinct candidate sets, and turn 2 returned only a clarification question with no table, chart, or figures. It also produced a merged comparison view in turn 1. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The user changes direction mid-conversation | 0 Pass1 Fail0 Not gradable 1 of 1 test case failedWhat happenedBlazeSQL failed to shift from the first-turn Texas filter to the broader all-states answer. In the second turn it dropped the filter but returned a state-by-state breakdown instead of the overall 12% rate; the export totals 100 orders and 12 cancellations, so the missing figure was available. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The user narrows what they just asked | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL preserved the earlier 90-day window when the user narrowed the request to Texas. The follow-up query kept the original date filter and added only the Texas filter, and the Texas export matched the Texas subset of the first export. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The next question refers to the previous answer | No published result | ||
Data Access Control3 scenarios · 2 with published Results
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| An excluded table is needed to answer the question | 0 Pass1 Fail0 Not gradable 1 of 1 test case failedWhat happenedBlazeSQL fails when the needed table is excluded. It never said it could not answer; instead it returned 4 from support tickets and then widened the search to pending confirmations, action log, and orders, while still never querying the excluded table. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Two users are given different data scope | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL kept the two scopes apart. The full-access account returned plus for the membership question, while the restricted account refused to look the customer up and showed no SQL or result table. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| An excluded column is asked for directly | No published result | ||
Question Answering4 scenarios · 3 with published Results
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| The answer is available in the connected data | Different test setPublished Result available, but not included in this comparison. | View Result → | |
| The connected data cannot answer the question | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL said it could not answer from the available data when asked how many 5-star reviews were received this month. It listed ten available tables, gave no number, and pointed to adding review data or using a related source instead. The key finding is that it declined rather than inventing a count. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The question uses a term the company defines itself | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL did not answer revenue as a bare number; it stated the definition it applied and showed the breakdown it used. In the captured Sep 2, 2026 run, it reported order revenue, subscription revenue, refunds, and an adjusted net revenue total, with the order line explicitly marked as excluding cancelled orders. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The user's wording doesn't match how the values are stored | No published result | ||
Reporting3 scenarios · 3 with published Results
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| The report is run again after the data has changed | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL kept the saved report current after new rows were added. The fixed-date save rose from 52 to 55, and the relative 30-day save rose from 162 to 164. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The user wants several findings in one place | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL passed this one-report task. The combined table kept the earlier revenue, category, and cancellation answers intact, with the same values and bases, and brought all five metrics into one report. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The user wants to keep an answer they just got | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL saves an answer and its query, then reopens the saved item with the same three rows under the saved name. The saved item appears in a separate Saved area, and the reopened table matches the earlier chat answer. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Training4 scenarios · 3 with published Results
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| The database's names are cryptic and the user documents them | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL used the user’s documentation to answer from the cryptic table and column. It reported 6 returns currently awaiting manual fraud review, split across 5 approved and 1 received. The stored note was applied to returns.flg_x2, and the follow-up answer came from that documented column. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The user sets a standing rule for all future answers | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL kept the standing rule across future questions. After the rule was set once, the later revenue answer returned 51,848.02 and the later order-count answer returned 32, and both generated queries still filtered out cancelled orders. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The user supplies example questions and the queries they trust | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL generalized the supplied question-query pair in this case. It first asked what counted as successful, then after the example query was added it returned 42 for both the repeated question and a reworded one; the same-session reference value was 42. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The agent used the wrong definition and the user corrects it | No published result | ||
Reading these Results
Published evidence and test coverage answer different questions.
Which scenarios have a Result?
A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.
What does each Result cover?
Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.
Inventory is not testing progress
The 28 scenarios describe this benchmark’s scope. They are not an assumed applicability or test-coverage denominator for BlazeSQL.