Scenario in benchmark · AI Database Agents · Version 1

The answer is available in the connected data

Whether the question can be answered from the connected data and the agent returns the correct result when it can.

Capability: Question Answering4 pinned test cases per toolNeutral global definition →

What we check

  1. Whether the agent answers from the connected data when the information is available.
  2. Whether the reply matches the correct result supported by the data.
  3. Whether the agent stays grounded in the database rather than guessing or inventing an answer.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Different test set · 6
Anomaly AIDifferent test setAnswered the four database questions with the reference count, month-to-month difference, order-value total and tied top products.View Result →

The corresponding reference values were 24 orders, 9 more orders, $14,403.87 USD and 6 units for each joint leader.

Published on a different test set, so its counts are not compared here.

AskYourDatabaseDifferent test setAnswered three connected-data questions correctly, but missed the 30-day order count.View Result →

It returned 24 instead of 25 for that lookup, while it got the month difference, August total, and top product by units right.

Published on a different test set, so its counts are not compared here.

BlazeSQLDifferent test setAnswered the lookup, month-comparison, and monthly-sum questions correctly, but missed the top-product ranking question.View Result →

It returned 57 orders for the 30-day window, 17 more August orders than July, and 20,697.92 for August; on the ranking task it named the wrong product and gave 10 units.

Published on a different test set, so its counts are not compared here.

camelAIDifferent test setAnswered three of the four database questions as the reference queries did: 10 more orders in August than July, an August total of $14,403.87 across 28 orders, and a tie at 6 units for the top product.View Result →

Asked for orders in the last 30 days as of 2 September, it counted from 4 August instead of 3 August and gave 24, against the reference 26.

Published on a different test set, so its counts are not compared here.

DraxlrDifferent test setAnswered the connected-data questions with the required counts, total and a top-ranked product.View Result →

It returned 26 orders, an August–July difference of 10, and an August total of $14,403.87, against references of 26, 10 and $14,403.87. Its named product had the maximum 6 units, without mentioning the other product tied at 6.

Published on a different test set, so its counts are not compared here.

FutureSmart Database AgentDifferent test setAnswered the connected-data questions correctly in all four cases.View Result →

It returned 57 orders for 3 August to 2 September 2026, 17 more August orders than July, USD 20,697.92 for August orders, and Alder bookshelf, 5-tier with 9 units.

Published on a different test set, so its counts are not compared here.

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (5): AI for Database, Basedash, Definite, Dot, Querio.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
4
Disclosure
0 public · 4 withheld
Capabilities exercised here
Question Answering

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 pins a scenario rubric for this scenario. Benchmark methodology →

The answer is available in the connected data in AI Database Agents | AI Demos