Scenario in benchmark · AI Database Agents · Version 1

The report is run again after the data has changed

A saved report is rerun after the underlying data has changed.

Capability: Reporting1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the rerun preserves the original question rather than becoming a different report.
  2. Whether the refreshed result is recomputed against the changed data state.
  3. Whether the answer still means the same thing as the saved report, not a frozen past result.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 2
AskYourDatabaseFailed1 of 1 test case failedAnswered the relative-period order question, but no save control was found for rerunning the answer as a report after data changed.View Result →

The answer actions offered copying and PDF printing.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

FutureSmart Database AgentFailed1 of 1 test case failedFailed this scenario.View Result →

The pinned question returned 57 for the 30-day window, but the console sweep shows no way to save an answer as a report or rerun one, so the result cannot refresh after rows are added.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 4
Anomaly AIPassed1 of 1 test case passedRefreshed the saved report from 24 to 27 orders, matching the new database count of 27.View Result →

It preserved the original reporting period after three orders were added within it.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

BasedashPassed1 of 1 test case passedPreserved the requested period when the saved report was refreshed after data changed.View Result →

The order count rose from 23 to 26, the new reference count.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

BlazeSQLPassed1 of 1 test case passedKept the saved report current after new rows were added.View Result →

The fixed-date save rose from 52 to 55, and the relative 30-day save rose from 162 to 164.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

DraxlrPassed1 of 1 test case passedRefreshed the saved report after three orders were added, keeping the same period.View Result →

Both the report and the database count for that period moved from 26 to 29 orders.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (5): AI for Database, camelAI, Definite, Dot, Querio.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Reporting

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

The report is run again after the data has changed in AI Database Agents | AI Demos