Scenario in benchmark · AI Database Agents · Version 1
The user supplies example questions and the queries they trust
A user supplies verified question-and-query examples that should shape later answers.
What we check
- Whether the agent uses the supplied example to improve a later answer on the same topic.
- Whether the agent handles a rephrased later question consistently with the supplied example.
- Whether the behaviour shows generalisation beyond the exact wording of the original example.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
FutureSmart Database AgentFailed1 of 1 test case failedDid not generalise the trusted example pair.View Result →
The exact wording returned 42, while the reworded question returned 41 from a new delivery-date query; before training, the same question was refused.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Anomaly AIPassed1 of 1 test case passedUsed a trusted question-and-query example to answer both the original wording and a rephrased question.View Result →
Both answers returned the verified result of 42 orders, compared with 38 before training on a different definition of success.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
AskYourDatabasePassed1 of 1 test case passedUsed a supplied question-and-query example beyond the original wording.View Result →
The repeated and reworded questions each returned 42 orders, equal to the trusted query’s result.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
BasedashPassed1 of 1 test case passedPassed this training scenario: after a trusted question-and-query pair was saved as a verified model, it answered both the repeated question and a reworded version with 42.View Result →
The untrained chat had answered 39, and the verified model preview also returned 42.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
BlazeSQLPassed1 of 1 test case passedGeneralized the supplied question-query pair in this case.View Result →
It first asked what counted as successful, then after the example query was added it returned 42 for both the repeated question and a reworded one; the same-session reference value was 42.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
DotPassed1 of 1 test case passedUsed the supplied question-and-query pair for both the original question and a rephrasing, returning the verified result of 42 successful orders for each.View Result →
Before training, it asked for clarification and gave no count.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
DraxlrPassed1 of 1 test case passedGeneralized a trusted question-and-query example to a rephrased question.View Result →
After training, both phrasings returned 42, the trusted count, and the rephrased question used the trusted query’s logic.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (4): AI for Database, camelAI, Definite, Querio.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Training
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →