Scenario in benchmark · AI Database Agents · Version 1
Reported analytics match what actually happened
Whether reported analytics align with the underlying recorded activity.
What we check
- Whether the reported figures match the activity that actually occurred.
- Whether the result reflects the same underlying events that were recorded.
- Whether discrepancies between reported numbers and recorded events are exposed rather than treated as correct.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
BlazeSQLFailed1 of 1 test case failedDid not report analytics that matched what happened.View Result →
Its query review page listed only one of five questions and left out both deliberately unanswerable ones, with no answered-versus-unanswered breakdown.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
DotFailed1 of 1 test case failedDot's History message total rose from 49 to 58 (+9) after six messages were sent, rather than the expected 55; the three extra came from sessions Dot created about missing data and counted as the user's messages.View Result →
Both unanswered chats were kept with one message each, and filtering to Analysis still showed 58.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
AskYourDatabasePassed1 of 1 test case passedAskYourDatabase's reported counts matched what was run in the two accounts.View Result →
The first account showed 3 conversations and the second showed 2, and the messages list showed the five questions with timestamps; each count included an unanswered chat.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
FutureSmart Database AgentPassed1 of 1 test case passedCounted every question that was run, including the two the database could not answer, so the total did not drop unanswered work.View Result →
The dashboard shows 6 answered and 0 unanswered, and the query log lists the same six rows, including the two unanswerable questions and a separate hi check.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (7): AI for Database, Anomaly AI, Basedash, camelAI, Definite, Draxlr, Querio.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Analytics & Observability
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →