Scenario in benchmark · AI Database Agents · Version 1
The next question refers to the previous answer
A follow-up question refers back to the immediately preceding answer and keeps that referent unchanged.
What we check
- Whether the agent resolves the follow-up from the immediately preceding answer.
- Whether the same referent stays in force across the turn instead of being silently replaced or recomputed.
- Whether the agent answers using the prior answer’s context without needing the original question repeated.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
Anomaly AIPassed1 of 1 test case passedKept the follow-up tied to the previous answer's five customers.View Result →
It answered turn 2 without re-asking who they were, and the reply opened with the same set while the visible query fragment reused the earlier cancelled-order rule.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
AskYourDatabasePassed1 of 1 test case passedKept the follow-up tied to the previous answer.View Result →
Turn 1 returned five customers, and turn 2 used the same five without a clarifying question or a silent change in the set.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
BasedashPassed1 of 1 test case passedKept the previous answer’s set in a follow-up question.View Result →
It counted 86 orders for the same five customers without asking who they were.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
DotPassed1 of 1 test case passedKept the follow-up tied to the five customers in its previous answer.View Result →
It reported 86 recent orders for that group, without asking who they were or recalculating the spend ranking.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
DraxlrPassed1 of 1 test case passedKept the previous answer’s customer set intact in the follow-up.View Result →
It answered for the same five customers, did not ask who “they” were, and the second query rebuilt that same set before counting their last-90-day orders.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
FutureSmart Database AgentPassed1 of 1 test case passedHandled the follow-up for the same five customers returned before.View Result →
It returned those five names again and reported order counts of 10, 9, 8, 6, and 5, with a total of 38.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (5): AI for Database, BlazeSQL, camelAI, Definite, Querio.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Conversational Interaction
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →