Scenario in benchmark · AI Database Agents · Version 1
The user's wording doesn't match how the values are stored
A user’s term and the stored value mean the same thing, but use different words.
What we check
- Whether it maps a natural synonym to the stored value it represents.
- Whether it returns the matching records rather than treating the question as unanswered.
- Whether it treats the wording difference as a translation problem, not as missing data.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
DraxlrFailed1 of 1 test case failedDid not bridge the wording gap for the gift voucher request.View Result →
It searched for those literal words and presented an empty result as the answer.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
FutureSmart Database AgentFailed1 of 1 test case failedFailed to handle wording that did not match stored values.View Result →
It answered with no results and treated that emptiness as a fact about the stored orders; its SQL only searched literal variants of the user's wording, not the stored payment value. The evidence also notes a matching gift-card value exists, with 2 orders totaling $984.06.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Anomaly AIPassed1 of 1 test case passedInterpreted gift vouchers as gift-card payments and returned the matching orders.View Result →
It listed 14 orders against a reference count of 14, including two cancelled orders.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
AskYourDatabasePassed1 of 1 test case passedReturned 14 gift-card orders for a request about gift vouchers.View Result →
The table matched the reference answer's $16,806.08 total and status breakdown.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
BasedashPassed1 of 1 test case passedUnderstood the gift-voucher request and returned all 14 matching orders.View Result →
Their total was USD 16,806.08, the stated reference total, and two were cancelled.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
BlazeSQLPassed1 of 1 test case passedHandled the synonym and returned the matching orders.View Result →
It showed two orders with the gift card payment method, and the written reply named the same two orders.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
camelAIPassed1 of 1 test case passedFound the orders despite the wording mismatch: asked for orders paid with a gift voucher, it searched the payment method for both gift card and gift voucher and returned the 14 gift card orders, two of them cancelled, as its answer.View Result →
No empty result was presented.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (4): AI for Database, Definite, Dot, Querio.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Question Answering
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →