Scenario in benchmark · AI Database Agents · Version 1

The follow-up is ambiguous

A follow-up could refer to more than one plausible set from the previous turn.

Capability: Conversational Interaction1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the agent recognises that the follow-up could refer to more than one prior set.
  2. Whether it asks for clarification instead of selecting one referent implicitly.
  3. Whether it avoids treating one candidate set as established before the ambiguity is resolved.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 6
Anomaly AIFailed1 of 1 test case failedChose a comparison rather than clarifying an ambiguous follow-up.View Result →

It compared five customers on order count, total spend and average order.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

AskYourDatabaseFailed1 of 1 test case failedDid not ask for clarification when a follow-up was ambiguous.View Result →

The first answer created two distinct customer lists, and the second turn answered with a side-by-side comparison instead of asking which list was meant.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

BasedashFailed1 of 1 test case failedChose an interpretation of an ambiguous follow-up without asking for clarification.View Result →

It combined the two top-three customer groups into a five-row comparison table with explanatory bullets.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

camelAIFailed1 of 1 test case failedAnswered the ambiguous follow-up without clarifying its meaning.View Result →

It compared the customer rankings by order count and spending, describing where the lists overlapped and how purchasing patterns differed.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

DraxlrFailed1 of 1 test case failedDid not ask for clarification when the follow-up was ambiguous.View Result →

After the first reply produced two plausible customer sets, the second reply answered directly and the transcript shows no clarifying question, option prompt, or disambiguation before it.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

FutureSmart Database AgentFailed1 of 1 test case failedDid not ask for clarification when the follow-up was ambiguous.View Result →

Turn 2 named the two rankings it had chosen and repeated the same comparison instead of asking which was meant.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 1
BlazeSQLPassed1 of 1 test case passedAsked which comparison the customer meant instead of silently choosing one.View Result →

Turn 1 showed two distinct candidate sets, and turn 2 returned only a clarification question with no table, chart, or figures. It also produced a merged comparison view in turn 1.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (4): AI for Database, Definite, Dot, Querio.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Conversational Interaction

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

The follow-up is ambiguous in AI Database Agents | AI Demos