Scenario in benchmark · AI Database Agents · Version 1

The user narrows what they just asked

A follow-up adds a narrowing constraint to an earlier question instead of replacing it.

Capability: Conversational Interaction1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the agent preserves the earlier question’s subject when answering the follow-up.
  2. Whether the agent applies the added constraint to the existing request rather than replacing the request.
  3. Whether the agent avoids silently broadening, shifting, or resetting the question.
  4. Whether the reply reflects the narrower scope only, not the earlier broader scope alone.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 1
DraxlrFailed1 of 1 test case failedKept the original 90-day condition when the user narrowed the question to Texas, but returned no orders even though three qualified.View Result →

It added a Texas filter using the full state name rather than the two-letter abbreviation stored in the order system.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 5
Anomaly AIPassed1 of 1 test case passedPasses this narrowing-follow-up scenario.View Result →

The second turn adds Texas while keeping the earlier 90-day window, and the result shows only the three matching orders while the outside-window Texas order stays absent.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

AskYourDatabasePassed1 of 1 test case passedHandles this narrowing case: it keeps the earlier question’s date window and adds the new Texas constraint.View Result →

In the first turn, the app returned 59 orders in a 10-row preview plus a full CSV. In the follow-up, it kept the same window, returned 3 Texas orders, and those three all appear in the first turn’s 59-row CSV.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

BasedashPassed1 of 1 test case passedAdded the Texas restriction while retaining the earlier 90-day condition.View Result →

It returned the expected three orders, rather than the 18 Texas orders across all dates.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

BlazeSQLPassed1 of 1 test case passedPreserved the earlier 90-day window when the user narrowed the request to Texas.View Result →

The follow-up query kept the original date filter and added only the Texas filter, and the Texas export matched the Texas subset of the first export.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

FutureSmart Database AgentPassed1 of 1 test case passedPassed this narrowing scenario: it kept the earlier 90-day window and added the Texas filter, and its answer reported 7 orders shipped to Texas.View Result →

The first answer used a 10-row limit without ordering, and the follow-up table used placeholder column labels.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (5): AI for Database, camelAI, Definite, Dot, Querio.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Conversational Interaction

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

The user narrows what they just asked in AI Database Agents | AI Demos