Scenario in benchmark · AI Database Agents · Version 1

The user changes direction mid-conversation

A conversation changes direction, and the agent must answer the new scope rather than carry the old one forward.

Capability: Conversational Interaction1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the agent drops the earlier subject or filter when the user changes direction.
  2. Whether the answer reflects only the new scope introduced by the follow-up.
  3. Whether the agent avoids silently carrying forward the prior constraint into the new answer.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 3
BlazeSQLFailed1 of 1 test case failedFailed to shift from the first-turn Texas filter to the broader all-states answer.View Result →

In the second turn it dropped the filter but returned a state-by-state breakdown instead of the overall 12% rate; the export totals 100 orders and 12 cancellations, so the missing figure was available.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

DraxlrFailed1 of 1 test case failedDropped the Texas filter when asked about all states, but reported 20.04% rather than the overall rate of 21.75%.View Result →

It used an unweighted average of state percentages.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

FutureSmart Database AgentFailed1 of 1 test case failedWhen the user changed direction, FutureSmart Database Agent dropped the earlier filter but did not give the unfiltered result the scenario calls for.View Result →

In the run, it returned a 16-row by-state breakdown instead of the single overall cancellation rate, so the answer did not match the unfiltered result.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 4
Anomaly AIPassed1 of 1 test case passedDropped the Texas filter when the user broadened the question to all states.View Result →

It returned 21.75%—67 cancellations from 308 orders—the all-states reference, rather than carrying forward the Texas-only rate of 27.78%.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

AskYourDatabasePassed1 of 1 test case passedHandles the mid-conversation change by dropping the earlier Texas filter and answering with the broader all-states breakdown.View Result →

The second query has no state condition, and the 15-row table totals 308 orders and 67 cancellations; the Texas turn stays separate at 18 orders and 27.78%.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

BasedashPassed1 of 1 test case passedDropped the Texas filter when the user broadened the question to all states.View Result →

Its answer matched the database’s all-states reference: 21.75%, or 67 cancelled orders out of 308.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

DotPassed1 of 1 test case passedDropped the Texas filter when the next question asked about all states.View Result →

It gave 27.8% (5 of 18 orders) for Texas, then 21.8% (67 of 308 orders) across all states, and its query for the second answer carried no state filter.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (4): AI for Database, camelAI, Definite, Querio.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Conversational Interaction

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

The user changes direction mid-conversation in AI Database Agents | AI Demos