Scenario in benchmark · AI Database Agents · Version 1
The user sets a standing rule for all future answers
A user sets a standing rule that should continue to apply to later, unrelated answers unless it is overridden.
What we check
- Whether the agent applies the standing rule to later, unrelated questions without the user repeating it.
- Whether the rule still affects answers after the conversation moves on to a different topic.
- Whether the agent changes its behaviour only when the user overrides or replaces the rule.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
Anomaly AIPassed1 of 1 test case passedFollowed the cancelled-order exclusion on both later questions without a reminder.View Result →
It returned $151,064.66 for 2026 revenue and 88 September orders, matching the filtered database references; without the exclusion, the figures were $198,836.65 and 97.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
AskYourDatabasePassed1 of 1 test case passedPassed this scenario: a standing rule carried into later, unrelated answers.View Result →
The exclusion was entered in Documentation once, and both later replies used it in the generated query and in the answer text.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
BasedashPassed1 of 1 test case passedRespected the user's standing rule on both later questions without a reminder.View Result →
It excluded cancelled orders from revenue and order-count calculations, and stated the exclusion in each answer.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
BlazeSQLPassed1 of 1 test case passedKept the standing rule across future questions.View Result →
After the rule was set once, the later revenue answer returned 51,848.02 and the later order-count answer returned 32, and both generated queries still filtered out cancelled orders.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
DotPassed1 of 1 test case passedApplied a standing rule to exclude cancelled orders, saved once in its Notes, to two later questions asked in new chats without the rule restated.View Result →
It reported $151,065 of revenue over 225 orders this year and 88 orders last month, the figures with cancelled orders excluded, and both of its queries filtered out cancelled orders.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
DraxlrPassed1 of 1 test case passedCarried a standing exclusion rule into two later, unrelated questions without a reminder.View Result →
Its revenue and order-count answers were $151,064.66 and 88, matching the database totals with cancelled orders excluded.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
FutureSmart Database AgentPassed1 of 1 test case passedPassed this scenario.View Result →
The rule was set once, and both later answers still excluded cancelled orders in their SQL and wording.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (4): AI for Database, camelAI, Definite, Querio.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Training
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →