Scenario in benchmark · AI Database Agents · Version 1
Customer corrects information given earlier
A later correction supersedes earlier information for the rest of the interaction.
What we check
- Whether the later correction replaces the earlier detail in later turns.
- Whether the response reflects only the corrected information, not a mix of old and new details.
- Whether the agent does not continue reasoning from the superseded detail after the correction.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
Anomaly AIPassed1 of 1 test case passedUsed the customer's correction in its answer.View Result →
After reporting 18 orders for July 2026, it answered with 11 for June and updated the insight to the corrected period.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
AskYourDatabasePassed1 of 1 test case passedAnswered the corrected period only.View Result →
The July question returned 23 orders, and the follow-up June correction produced a new June query and 10 orders, with the thread ending there.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
BasedashPassed1 of 1 test case passedUsed the customer’s correction and returned the database’s June 2026 total of 11 orders.View Result →
It answered for June alone, rather than keeping July’s answer or combining the months.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
BlazeSQLPassed1 of 1 test case passedReplaced the earlier July answer with the corrected June answer.View Result →
In the visible exchange, the first turn returned 21 orders for July 2026, and the follow-up turn rebuilt the query for June 2026 and returned 20 orders, matching the corrected period only.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
camelAIPassed1 of 1 test case passedUsed the customer's corrected month instead of retaining the original month or combining the two.View Result →
After initially answering 18 orders for July 2026, it returned 11 for June 2026—the same 11 as the database reference.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
DotPassed1 of 1 test case passedUpdated its answer when the customer corrected July to June.View Result →
The corrected reply gave 11 June orders and the query limited the count to June; the earlier July turn was 18 against a reference of 19 and was not the graded one.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
DraxlrPassed1 of 1 test case passedHandled the follow-up correction by using the updated month instead of the original one.View Result →
The second reply said June 2026, and the second query was limited to June only; it did not combine July with June or keep July in place.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
FutureSmart Database AgentPassed1 of 1 test case passedAnswered the correction with June only.View Result →
It reported 20 orders, used June-only SQL, and did not carry forward the July answer.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (3): AI for Database, Definite, Querio.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Conversational Interaction
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →