Scenario in benchmark · AI Database Agents · Version 1

Someone needs to see what people have been asking

A usage view shows what people asked and attributes those questions correctly.

Capability: Analytics & Observability1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the usage view lists the questions that were asked.
  2. Whether each question is attributed to the source that asked it.
  3. Whether the view accounts for the full set of questions, not just a subset.
  4. Whether the totals or counts shown by the view match the underlying activity.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 1
BlazeSQLFailed1 of 1 test case failedDoes not show a complete view of what people have been asking.View Result →

Query review lists one request in the default view and still shows one user with the filter set to All, while the two chat rails split the five questions between the two accounts and one account's own chat history is shown.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 6
Anomaly AIPassed1 of 1 test case passedShowed the questions asked across two accounts and attributed them to the right account.View Result →

The owner's usage view contained five insights for five questions: three from the owner and two from the member.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

AskYourDatabasePassed1 of 1 test case passedShowed what people had asked and which account asked each question.View Result →

Its usage list contained five entries for the five questions asked.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

BasedashPassed1 of 1 test case passedLet an admin see what both accounts had asked and identify the askers.View Result →

Its Chats list contained the five questions sent, with no missing or extra questions.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

DotPassed1 of 1 test case passedDot's History listed every question with the account that asked it: six from the admin account and three from the standard account.View Result →

Filtered to the standard account, the analysis footer read 3 chats for its 3 questions; its Messages tile read 5 because it also counts Root proposal sessions Dot created itself.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

DraxlrPassed1 of 1 test case passedMade the questions visible and attributable to the people who asked them.View Result →

The admin could see all five test chats in the sidebar and identify which account asked each question.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

FutureSmart Database AgentPassed1 of 1 test case passedShows what people have been asking: its Activity view reports 5 of 5 shown for a split between two people.View Result →

Three questions come from one account and two from the other, and the query log separately names the account for the net-promoter-score question.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (4): AI for Database, camelAI, Definite, Querio.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Analytics & Observability

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

Someone needs to see what people have been asking in AI Database Agents | AI Demos