Tool in benchmark · Version 1

FutureSmart Database Agent in AI Database Agents

Scenario-level performance from current published Results.

28 scenarios with published Results · 28 scenarios in the benchmark

How FutureSmart Database Agent performed

Open a capability to explore its scenarios. Each row reports the test set in its published Result; counts are not combined into an overall score.

Analytics & Observability4 scenarios · 4 with published Results
ScenarioPublished outcomesTest coverageResult
A user reports a bad answer and the operator has to find it
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

FutureSmart Database Agent can find a complained-about exchange from a vague clue. The query log shows the target row with its chat and session IDs among five queries, and the opened Activity page shows the same exchange read-only with the question, complete answer, and share-link attribution.

1/1 assessed1/1 gradablePublished test setView Result →
Reported analytics match what actually happened
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

FutureSmart Database Agent counted every question that was run, including the two the database could not answer, so the total did not drop unanswered work. The dashboard shows 6 answered and 0 unanswered, and the query log lists the same six rows, including the two unanswerable questions and a separate hi check.

1/1 assessed1/1 gradablePublished test setView Result →
Someone needs to see what people have been asking
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

FutureSmart Database Agent shows what people have been asking: its Activity view reports 5 of 5 shown for a split between two people. Three questions come from one account and two from the other, and the query log separately names the account for the net-promoter-score question.

1/1 assessed1/1 gradablePublished test setView Result →
The operator needs to find the questions the agent is failing on
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

FutureSmart Database Agent failed this scenario: the failures view showed zero failures and no failing questions. In the same window, Input 3 and Input 5—the complaints question and the net-promoter-score question—were both marked answered in the query log, so the questions it could not answer were not surfaced as failures.

1/1 assessed1/1 gradablePublished test setView Result →
Answer Presentation4 scenarios · 4 with published Results
ScenarioPublished outcomesTest coverageResult
The answer is a plain fact or a short list
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

FutureSmart Database Agent answered this case in plain prose rather than only as a chart or one-cell table. It said there were 100 orders in total, and the nearby count grid repeated the same number.

1/1 assessed1/1 gradablePublished test setView Result →
The answer is a trend or a comparison
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

FutureSmart Database Agent did not present the answer as a trend visual. It showed monthly counts in Table view with Chart available but not selected, and the month labels were raw timestamps.

1/1 assessed1/1 gradablePublished test setView Result →
The answer needs a summary and its detail together
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

FutureSmart Database Agent passes this scenario. It gave the August total (USD 20,253.47) and broke it down by all seven product categories in a paged table and a bar chart.

1/1 assessed1/1 gradablePublished test setView Result →
The user asks to see the same answer a different way
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

FutureSmart Database Agent failed to keep the same answer when asked for a different presentation. The follow-up returned the same table and said it could not render a chart, while an earlier turn in the same conversation showed a bar chart for the same data; the later turn's values also differed from the earlier chart.

1/1 assessed1/1 gradablePublished test setView Result →
Conversational Interaction6 scenarios · 6 with published Results
ScenarioPublished outcomesTest coverageResult
Customer corrects information given earlier
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

FutureSmart Database Agent answered the correction with June only. It reported 20 orders, used June-only SQL, and did not carry forward the July answer.

1/1 assessed1/1 gradablePublished test setView Result →
The agent offers what to ask next
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

FutureSmart Database Agent offered two follow-up questions after its August 2026 revenue answer, and both were specific to that answer and answerable from the database. The chips ask about August versus July and the highest-revenue month so far.

1/1 assessed1/1 gradablePublished test setView Result →
The follow-up is ambiguous
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

FutureSmart Database Agent did not ask for clarification when the follow-up was ambiguous. Turn 2 named the two rankings it had chosen and repeated the same comparison instead of asking which was meant.

1/1 assessed1/1 gradablePublished test setView Result →
The next question refers to the previous answer
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

FutureSmart Database Agent handled the follow-up for the same five customers returned before. It returned those five names again and reported order counts of 10, 9, 8, 6, and 5, with a total of 38.

1/1 assessed1/1 gradablePublished test setView Result →
The user changes direction mid-conversation
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

When the user changed direction, FutureSmart Database Agent dropped the earlier filter but did not give the unfiltered result the scenario calls for. In the run, it returned a 16-row by-state breakdown instead of the single overall cancellation rate, so the answer did not match the unfiltered result.

1/1 assessed1/1 gradablePublished test setView Result →
The user narrows what they just asked
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

FutureSmart Database Agent passed this narrowing scenario: it kept the earlier 90-day window and added the Texas filter, and its answer reported 7 orders shipped to Texas. The first answer used a 10-row limit without ordering, and the follow-up table used placeholder column labels.

1/1 assessed1/1 gradablePublished test setView Result →
Data Access Control3 scenarios · 3 with published Results
ScenarioPublished outcomesTest coverageResult
An excluded column is asked for directly
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

FutureSmart Database Agent could not be executed for this scenario because it cannot exclude a single column. Its only exclusion control works on whole tables on both connections, and no other settings screen offers column-level control. No question was asked, so the agent's handling of a direct request for an excluded column was not observed.

1/1 assessed1/1 gradablePublished test setView Result →
An excluded table is needed to answer the question
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

FutureSmart Database Agent failed the excluded-table scenario. It said it could not answer, but the chat trace shows it generated SQL, read the excluded returns table, and showed a 0 result.

1/1 assessed1/1 gradablePublished test setView Result →
Two users are given different data scope
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

FutureSmart Database Agent failed to keep the two accounts' answers within their own data scopes. The open account answered plus, while the restricted account answered from another table, gave bedding_and_bath, and its notice said it read the excluded table.

1/1 assessed1/1 gradablePublished test setView Result →
Question Answering4 scenarios · 4 with published Results
ScenarioPublished outcomesTest coverageResult
The answer is available in the connected dataDifferent test setPublished Result available, but not included in this comparison.View Result →
The connected data cannot answer the question
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

FutureSmart Database Agent fails when the connected data does not contain the answer. It answered the question with a count from its activity log, returned 0, and did not say it could not answer from the available data.

1/1 assessed1/1 gradablePublished test setView Result →
The question uses a term the company defines itself
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

FutureSmart Database Agent answered a company-defined revenue question by stating the definition it used instead of giving only a bare number. The answer block and generated SQL use the same 90-day, not-refunded definition, and the capture shows the matching USD total.

1/1 assessed1/1 gradablePublished test setView Result →
The user's wording doesn't match how the values are stored
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

FutureSmart Database Agent failed to handle wording that did not match stored values. It answered with no results and treated that emptiness as a fact about the stored orders; its SQL only searched literal variants of the user's wording, not the stored payment value. The evidence also notes a matching gift-card value exists, with 2 orders totaling $984.06.

1/1 assessed1/1 gradablePublished test setView Result →
Reporting3 scenarios · 3 with published Results
ScenarioPublished outcomesTest coverageResult
The report is run again after the data has changed
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

FutureSmart Database Agent failed this scenario. The pinned question returned 57 for the 30-day window, but the console sweep shows no way to save an answer as a report or rerun one, so the result cannot refresh after rows are added.

1/1 assessed1/1 gradablePublished test setView Result →
The user wants several findings in one place
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

FutureSmart Database Agent failed to keep the assembled findings consistent. It kept revenue at USD 20,253.47, but the combined report changed the category answer from storage_and_organization to outdoor_and_patio and the cancellation rate from 12.0% to 13.19%.

1/1 assessed1/1 gradablePublished test setView Result →
The user wants to keep an answer they just got
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

FutureSmart Database Agent did not let a user save an answer and reopen it later. The answer view showed viewing, filtering, charting, CSV export, thumbs, and copy controls, but no saved-answer destination.

1/1 assessed1/1 gradablePublished test setView Result →
Training4 scenarios · 4 with published Results
ScenarioPublished outcomesTest coverageResult
The agent used the wrong definition and the user corrects it
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

FutureSmart Database Agent kept the corrected revenue definition on the repeated question and the reworded one. After the customer gave the rule in chat, both later answers excluded cancelled orders and returned USD 51,848.02 from one row.

1/1 assessed1/1 gradablePublished test setView Result →
The database's names are cryptic and the user documents them
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

FutureSmart Database Agent answered the follow-up from the documented table and column. The follow-up query used flg_x2 = true on returns and returned 7; the training turn showed 7 false rows and 7 true rows.

1/1 assessed1/1 gradablePublished test setView Result →
The user sets a standing rule for all future answers
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

FutureSmart Database Agent passed this scenario. The rule was set once, and both later answers still excluded cancelled orders in their SQL and wording.

1/1 assessed1/1 gradablePublished test setView Result →
The user supplies example questions and the queries they trust
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

FutureSmart Database Agent did not generalise the trusted example pair. The exact wording returned 42, while the reworded question returned 41 from a new delivery-date query; before training, the same question was refused.

1/1 assessed1/1 gradablePublished test setView Result →

Reading these Results

Published evidence and test coverage answer different questions.

Publication availability

Which scenarios have a Result?

A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.

Test coverage

What does each Result cover?

Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.

Scenario scope

Inventory is not testing progress

The 28 scenarios describe this benchmark’s scope. They are not an assumed applicability or test-coverage denominator for FutureSmart Database Agent.

FutureSmart Database Agent in AI Database Agents | AI Demos