Scenario in benchmark · AI Database Agents · Version 1

The connected data cannot answer the question

A user asks for information that the connected database does not hold.

Capability: Question Answering1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the agent says the question cannot be answered from the available data.
  2. Whether the agent avoids inventing a factual answer or number.
  3. Whether the agent does not present an unrelated substitute as if it answered the question.
  4. Whether the agent keeps the response tied to the missing data condition rather than pretending the database contains it.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 1
FutureSmart Database AgentFailed1 of 1 test case failedFails when the connected data does not contain the answer.View Result →

It answered the question with a count from its activity log, returned 0, and did not say it could not answer from the available data.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 6
Anomaly AIPassed1 of 1 test case passedSaid when the connected data could not answer a question.View Result →

Asked about five-star reviews, it explained that the data held orders and support tickets but no reviews or star ratings, and gave no count.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

AskYourDatabasePassed1 of 1 test case passedCould not answer the question from the connected data.View Result →

It said the schema it could see had no review or rating data, asked where that data is stored, and returned no number.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

BasedashPassed1 of 1 test case passedDeclined to give a count of 5-star reviews for October 2026 from the connected data.View Result →

It described the count as unavailable rather than zero.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

BlazeSQLPassed1 of 1 test case passedSaid it could not answer from the available data when asked how many 5-star reviews were received this month.View Result →

It listed ten available tables, gave no number, and pointed to adding review data or using a related source instead. The key finding is that it declined rather than inventing a count.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

camelAIPassed1 of 1 test case passedSaid it could not answer how many 5-star reviews came in this month, because the connected database has no review data, and gave no number.View Result →

It listed the database's tables before replying, and none of the ten covers reviews or ratings.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

DraxlrPassed1 of 1 test case passedIdentified missing review data and gave no count for this month’s five-star reviews.View Result →

It asked which part of the database should represent those reviews.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (4): AI for Database, Definite, Dot, Querio.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Question Answering

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

The connected data cannot answer the question in AI Database Agents | AI Demos