Scenario in benchmark · AI Database Agents · Version 1

The database's names are cryptic and the user documents them

A user documents cryptic table or column names so later questions can be interpreted correctly.

Capability: Training1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the agent uses the supplied documentation when resolving a later question about the cryptic names.
  2. Whether the answer is grounded in the documented table or column rather than in a guess from the raw name.
  3. Whether the same documentation still works when the later question is phrased differently.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Passed · 7
Anomaly AIPassed1 of 1 test case passedUsed user-written documentation to answer a question about cryptically named data.View Result →

It counted 7 returns flagged for manual fraud review, matching the reference count of 7.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

AskYourDatabasePassed1 of 1 test case passedUsed buyer-written documentation to reach the requested data.View Result →

It counted 7 returns flagged for manual fraud review, matching the reference count of 7.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

BasedashPassed1 of 1 test case passedAfter a one-line description of a cryptically named column was added to its AI context, Basedash answered that 7 returns are flagged for manual fraud review.View Result →

Its query counted returns on exactly the documented flag, and 7 is the same count computed directly on the test database.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

BlazeSQLPassed1 of 1 test case passedUsed the user’s documentation to answer from the cryptic table and column.View Result →

It reported 6 returns currently awaiting manual fraud review, split across 5 approved and 1 received. The stored note was applied to returns.flg_x2, and the follow-up answer came from that documented column.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

DotPassed1 of 1 test case passedAfter a note describing the cryptically named flag column was saved, Dot answered that 7 returns are flagged for manual fraud review, counting exactly the documented column on the returns table; the reference count is 7.View Result →

Without the note it had reached the same count but asked whether it had picked the right flag.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

DraxlrPassed1 of 1 test case passedUsed the supplied documentation to interpret a cryptic column name and select the right data for the question.View Result →

It returned 7 returns flagged for manual fraud review, matching the reference count of 7.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

FutureSmart Database AgentPassed1 of 1 test case passedAnswered the follow-up from the documented table and column.View Result →

The follow-up query used flg_x2 = true on returns and returned 7; the training turn showed 7 false rows and 7 true rows.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (4): AI for Database, camelAI, Definite, Querio.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Training

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

The database's names are cryptic and the user documents them in AI Database Agents | AI Demos