Scenario in benchmark · AI Database Agents · Version 1
The answer needs a summary and its detail together
When an answer needs both an overall summary and supporting detail, the system presents both together.
What we check
- Whether it presents both the overall summary and the supporting detail in one reply.
- Whether it keeps the summary and the supporting detail aligned.
- Whether it uses forms suited to each part instead of forcing a single form.
- Whether it still answers the same underlying question when both parts are requested together.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
BlazeSQLFailed1 of 1 test case failedFailed to give the overall figure together with the breakdown.View Result →
It first asked which dimension to use, then showed category-level query, results, highlights, and CSV export, but none stated the month total.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Anomaly AIPassed1 of 1 test case passedPresented a headline figure alongside its breakdown.View Result →
It reported that August revenue fell 19.14% from July and accompanied the answer with a monthly comparison and a category chart.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
AskYourDatabasePassed1 of 1 test case passedGave a headline figure and a breakdown in forms suited to each, so it met the scenario.View Result →
It returned a revenue summary table for the total, a category table for the breakdown, and takeaways naming the driver; there was no chart.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
BasedashPassed1 of 1 test case passedPasses this presentation case.View Result →
It gave a headline figure, a month-comparison table, and a separate category breakdown whose contributions add to the stated change.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
DotPassed1 of 1 test case passedDid combine a headline figure with its breakdown.View Result →
It gave an August sales sentence with a monthly chart, then a contribution chart and driver sentences; Furniture was the main driver.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
DraxlrPassed1 of 1 test case passedAnswered a question about last month's revenue with a summary and its breakdown together in one reply.View Result →
It stated in prose that August revenue was $14,403.87, down $3,858.77 from July's $18,262.64, and set out each of seven categories' August and July revenue and change in a table, led by furniture's $6,562.19 drop.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
FutureSmart Database AgentPassed1 of 1 test case passedPasses this scenario.View Result →
It gave the August total (USD 20,253.47) and broke it down by all seven product categories in a paged table and a bar chart.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (4): AI for Database, camelAI, Definite, Querio.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Answer Presentation
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →