Scenario in benchmark · AI Database Agents · Version 1
The operator needs to find the questions the agent is failing on
A reviewer can locate failed or unanswered questions, and they are surfaced as failures rather than ordinary successful answers.
What we check
- Whether failed or unanswered questions can be found from the recorded activity.
- Whether those cases are surfaced as failures rather than as ordinary answered questions.
- Whether the system makes the failure state distinguishable from a successful answer.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
AskYourDatabaseFailed1 of 1 test case failedDid not surface unanswered questions as failures.View Result →
It placed two questions it could not answer in the same Messages list style as three it answered.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
BlazeSQLFailed1 of 1 test case failedFailed to surface unanswered questions as failures.View Result →
In the recording, Query review showed one ordinary row marked 'Not verified' and no failures or unanswered view appeared anywhere in the sections shown. The social-media complaints question appeared in Chat with a normal answer, and the other unanswerable question is not shown in the admin view.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
FutureSmart Database AgentFailed1 of 1 test case failedFailed this scenario: the failures view showed zero failures and no failing questions.View Result →
In the same window, Input 3 and Input 5—the complaints question and the net-promoter-score question—were both marked answered in the query log, so the questions it could not answer were not surfaced as failures.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (8): AI for Database, Anomaly AI, Basedash, camelAI, Definite, Dot, Draxlr, Querio.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Analytics & Observability
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →