Scenario in benchmark · AI Database Agents · Version 1
A user reports a bad answer and the operator has to find it
A bad-answer report is used to locate the specific complained-about exchange for inspection.
What we check
- Whether a specific complained-about exchange can be located from a bad-answer report.
- Whether the system uses the report's limited context to find one exact exchange rather than only a summary of failures.
- Whether the located exchange is available for inspection.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
AskYourDatabasePassed1 of 1 test case passedLet the admin locate and inspect the reported exchange about social media complaints.View Result →
The reply said it could not access social media data and offered to count support tickets instead.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
BasedashPassed1 of 1 test case passedMade the complained-about exchange findable and inspectable.View Result →
The chat list led to the question and full answer, and also opened two chats from another member.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
DotPassed1 of 1 test case passedCan find a complained-about exchange from a rough time and the member who asked it, then open it for inspection.View Result →
It locates the chat in History by user and time, narrows it with a membership search, and shows the question and reply.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
DraxlrPassed1 of 1 test case passedLet the admin find and inspect the reported exchange in saved chats.View Result →
The opened conversation exposed the original question and response, with the SQL query and its explanation available alongside them.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
FutureSmart Database AgentPassed1 of 1 test case passedCan find a complained-about exchange from a vague clue.View Result →
The query log shows the target row with its chat and session IDs among five queries, and the opened Activity page shows the same exchange read-only with the question, complete answer, and share-link attribution.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (6): AI for Database, Anomaly AI, BlazeSQL, camelAI, Definite, Querio.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Analytics & Observability
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →