Scenario in benchmark · AI Database Agents · Version 1

The agent used the wrong definition and the user corrects it

A user corrects an earlier definition, and the agent must apply that correction consistently in later questions and rephrasings.

Capability: Training1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the agent uses the corrected definition on the next related question.
  2. Whether the corrected definition still holds when the later question is phrased differently.
  3. Whether the agent keeps the correction in force instead of reverting to the earlier wrong rule.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Passed · 5
AskYourDatabasePassed1 of 1 test case passedKept the corrected definition on both the repeated and reworded questions.View Result →

Each later answer gave the reference revenue total of $23,322.52, compared with $23,597.02 before training.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

BasedashPassed1 of 1 test case passedApplied the revenue correction saved in its AI context to both the repeated question and a reworded one, each asked in a new chat.View Result →

Both returned $23,469.02 across 42 non-cancelled orders, the total with cancelled orders left out, against $39,806.75 across 57 orders if they were counted.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

BlazeSQLPassed1 of 1 test case passedKept the corrected revenue rule across both follow-up questions, and both answers matched the reference total of $23,469.02 from 42 orders.View Result →

The saved note was already on the orders table, and the later queries excluded cancelled orders and ended the window before September 3.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

DraxlrPassed1 of 1 test case passedRetained the user’s corrected metric definition for both a repeated question and a differently worded one.View Result →

Both revenue answers were $23,194.52, the same as the reference total.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

FutureSmart Database AgentPassed1 of 1 test case passedKept the corrected revenue definition on the repeated question and the reworded one.View Result →

After the customer gave the rule in chat, both later answers excluded cancelled orders and returned USD 51,848.02 from one row.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (6): AI for Database, Anomaly AI, camelAI, Definite, Dot, Querio.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Training

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

The agent used the wrong definition and the user corrects it in AI Database Agents | AI Demos