Scenario in benchmark · AI Database Agents · Version 1

The question uses a term the company defines itself

A question uses a business term whose meaning is set by the company rather than by the database schema.

Capability: Question Answering1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the agent recognises that the term may have a company-defined meaning rather than a schema-defined one.
  2. Whether the agent asks for, states, or applies the relevant business definition instead of treating the term as unambiguous.
  3. Whether the agent keeps that chosen meaning consistent in the answer it gives.
  4. Whether the agent avoids presenting a bare factual answer when the term still needs interpretation.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Passed · 7
Anomaly AIPassed1 of 1 test case passedHandled the business-defined revenue question by stating the definition it used instead of giving a bare number.View Result →

It returned booked revenue, excluded cancelled orders, did not deduct refunds, used current statuses, and offered a follow-up to calculate revenue net of refunds.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

AskYourDatabasePassed1 of 1 test case passedStated its chosen definition of revenue rather than treating the company-defined term as unambiguous.View Result →

It returned $24,944.28 based on order totals with cancelled orders excluded.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

BasedashPassed1 of 1 test case passedAnswered the revenue question with an explicit definition rather than a bare number.View Result →

It also stated the period and time zone and distinguished totals before and after refunds.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

BlazeSQLPassed1 of 1 test case passedDid not answer revenue as a bare number; it stated the definition it applied and showed the breakdown it used.View Result →

In the captured Sep 2, 2026 run, it reported order revenue, subscription revenue, refunds, and an adjusted net revenue total, with the order line explicitly marked as excluding cancelled orders.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

camelAIPassed1 of 1 test case passedStated the revenue definition it applied instead of giving a bare number: $23,235.70, from 42 delivered orders less one $233.32 refund, with 15 cancelled orders excluded.View Result →

The same 90 days support three defensible figures, $39,806.75, $23,469.02 and $23,235.70, and its figure is the one its stated definition picks out.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

DraxlrPassed1 of 1 test case passedAnswered that revenue over the 90 days to September 2, 2026 was $39,806.75 and said in the same answer that it used all order statuses.View Result →

Its explanation and generated query match that stated choice, which counts cancelled orders as revenue.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

FutureSmart Database AgentPassed1 of 1 test case passedAnswered a company-defined revenue question by stating the definition it used instead of giving only a bare number.View Result →

The answer block and generated SQL use the same 90-day, not-refunded definition, and the capture shows the matching USD total.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (4): AI for Database, Definite, Dot, Querio.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Question Answering

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

The question uses a term the company defines itself in AI Database Agents | AI Demos