Can Draxlr answer questions using connected data?
Draxlr answered the connected-data questions with the required counts, total and a top-ranked product. It returned 26 orders, an August–July difference of 10, and an August total of $14,403.87, against references of 26, 10 and $14,403.87. Its named product had the maximum 6 units, without mentioning the other product tied at 6.
4 of 4 test cases passed
Every test case in this scenario has a result.
The 4 test cases
Each test case is judged on its own: Pass, Fail, Not gradable, or Untested. The scenario result above is based on these test cases.
Order count in a defined periodPassEvidence
“As of September 2, 2026, how many orders were placed in the last 30 days?”
26 orders were placed in the last 30 days as of September 2, 2026. This counts orders from August 3 through September 2, 2026, using UTC timestamps.
Transcribed from Proof 1 · Screen recording at 0:06
Everything the tool returned, in order (2)
- Query behind the order count · not part of the answer
SELECT COUNT(*) AS orders_placed_last_30_days FROM public.orders WHERE placed_at >= (TIMESTAMPTZ '2026-09-02 00:00:00+00' - INTERVAL '30 days') AND placed_at < (TIMESTAMPTZ '2026-09-02 00:00:00+00' + INTERVAL '1 day')
- Answer
The final response, printed above.
| ✓ | Expected: 26 orders.Found: It returned 26 orders; the reference count for the required window is 26. |
| ✓ | Expected: 3 August–2 September 2026 inclusive in UTC, resolved from the as-of date and stated or evident in the answer.Found: It stated 3 August–2 September 2026 inclusive in UTC in both the answer and the explanation. |
Draxlr's query counted precisely the required 3 August–2 September 2026 UTC window. The 26-order answer is supported by the same calculation the requirement calls for, and the stated period uses the question's as-of date.
Order-count difference between two monthsPassEvidence
“As of September 2, 2026, how many more orders were placed in August 2026 than in July 2026?”
August 2026 had 10 more orders than July 2026. This uses the full calendar months for both July and August 2026.
Transcribed from Proof 1 · Screen recording at 0:06
Everything the tool returned, in order (2)
- Query behind the monthly comparison · not part of the answer
SELECT COUNT(*) FILTER ( WHERE placed_at >= TIMESTAMPTZ '2026-08-01 00:00:00+00' AND placed_at < TIMESTAMPTZ '2026-09-01 00:00:00+00' ) - COUNT(*) FILTER ( WHERE placed_at >= TIMESTAMPTZ '2026-07-01 00:00:00+00' AND placed_at < TIMESTAMPTZ '2026-08-01 00:00:00+00' ) AS more_orders_in_august FROM public.orders WHERE placed_at >= TIMESTAMPTZ '2026-07-01 00:00:00+00' AND placed_at < TIMESTAMPTZ '2026-09-01 00:00:00+00' - Answer
The final response, printed above.
| ✓ | Expected: 10 more orders in August 2026 than July 2026: 28 versus 18.Found: It returned 10 more orders; the reference difference is 10, from August's 28 and July's 18. |
| ✓ | Expected: State that August 2026 had more orders than July 2026: 28 versus 18.Found: It explicitly identified August 2026 as higher than July 2026; the reference counts are 28 and 18, respectively. |
Draxlr compared the full UTC calendar months without excluding orders by status and identified August as higher. Its calculation used the required month boundaries and order scope, satisfying both the numerical and direction requirements.
Monthly counts omitted. The August–July answer gives the difference without the two monthly counts, 28 and 18, so its arithmetic cannot be checked from the answer alone.
Comparison assumptions unstated. The August–July answer does not identify UTC or say cancelled orders are included. Using India-time months would give a difference of 9.
Total order value in a calendar monthPassEvidence
“As of September 2, 2026, what was the combined total amount of all orders placed in August 2026?”
As of September 2, 2026, the combined total amount of all orders placed in August 2026 was $14,403.87.
Transcribed from Proof 1 · Screen recording at 0:06
Everything the tool returned, in order (2)
- Query behind the August total · not part of the answer
SELECT COALESCE(SUM(total_amount), 0) AS combined_total_amount FROM public.orders WHERE placed_at >= TIMESTAMPTZ '2026-08-01 00:00:00+00' AND placed_at < TIMESTAMPTZ '2026-09-01 00:00:00+00' AND placed_at <= TIMESTAMPTZ '2026-09-02 23:59:59+00'
- Answer
The final response, printed above.
| ✓ | Expected: $14,403.87 across 28 orders placed in August 2026.Found: It returned $14,403.87; the reference total is $14,403.87 across 28 August orders. |
| ✓ | Expected: State August 2026 and include all orders, without excluding any by status.Found: It named August 2026 and all orders. Its query included 6 cancelled orders worth $5,608.42; excluding them would give $8,795.45. |
The request called for August's total without dropping orders by status. Draxlr's query included cancelled orders and used the required month boundaries; the reference total includes those orders too, so the scope and amount satisfy the request.
Total scope left implicit. The August-total answer repeats the request's all-orders scope without explicitly mentioning cancelled orders or stating the order count.
Total time zone unstated. The August-total query uses UTC month boundaries, which the answer does not mention. An India-time August would total $13,232.07.
Rounded chat card. The August-total chat card rounds the amount to 14.4K; the answer text and the Visualization and Result tabs give the exact $14,403.87.
Top product by units ordered in a monthPassEvidence
“As of September 2, 2026, which product had the most units ordered in August 2026, and how many units?”
The top product was Brass picture frame, 8x10 with 6 units ordered in August 2026.
Transcribed from Proof 1 · Screen recording at 0:06
Everything the tool returned, in order (2)
- Query behind the top-product result · not part of the answer
SELECT p.id AS product_id, p.name AS product_name, p.sku, SUM(CAST(oi.quantity AS NUMERIC)) AS units_ordered FROM public.order_items oi JOIN public.orders o ON o.id = oi.order_id JOIN public.products p ON p.id = oi.product_id WHERE o.placed_at >= TIMESTAMPTZ '2026-08-01 00:00:00+00' AND o.placed_at < TIMESTAMPTZ '2026-09-01 00:00:00+00' GROUP BY p.id, p.name, p.sku ORDER BY units_ordered DESC, p.name ASC LIMIT 1
- Answer
The final response, printed above.
| ✓ | Expected: The 8-by-10 brass picture frame or 24-inch round wall mirror, tied at 6 units.Found: It named the 8-by-10 brass picture frame; the reference leaders are that frame and the 24-inch round wall mirror, tied at 6 units. |
| ✓ | Expected: 6 units ordered in August 2026.Found: It reported 6 units, equal to the reference maximum of 6. |
Either product tied for the maximum meets the top-product requirement, so naming the brass picture frame qualifies. Its 6 units equal the reference maximum of 6.
Tie not disclosed. The top-product answer names the 8-by-10 brass picture frame as the single leader and does not mention the 24-inch round wall mirror, which also had 6 units.
Alphabetical tie-break. For the August top-product result, the query returns one product and breaks ties alphabetically by name, rather than by a business criterion.
Ranking scope unstated. The top-product answer does not state that August is measured in UTC or that units from cancelled orders are included.
Configuration and setup
How this tool was set up for the run and what the test needed in place. Each row is a fact from the run's records; a fact the records do not hold is left out, not guessed.
How this scenario is graded
How we decide Pass, Fail and Not gradable. The same rules apply to every tool tested on this scenario.
Each test case gets one result per tool: Pass, Fail or Not gradable. A test case we haven't run yet shows Untested. There are no partial results.
- Pass: the tool did everything the test expected, and nothing it said contradicts the correct answer.
- Fail: at least one expected behaviour clearly didn't happen; the test case's result says which one and quotes the tool.
- Not gradable: our evidence couldn't settle the outcome (for example a record we needed is missing). It is never counted as a fail, and the test case's result says what's missing.
Where this sits in the benchmark
This page is one cell of a larger study: one tool, one scenario. Only this benchmark's frame appears here.
| Level | Name | Scope |
|---|---|---|
| Benchmark | AI Database Agents → | 28 scenarios · 11 tools |
| Capability | Question Answering → | |
| Scenario | The answer is available in the connected data → | |
| Tool | Draxlr → |
Global scenario definition → · Global capability definition → · Draxlr product page →
History of this result
What has happened to this result since it was first published. Runs and grades are never overwritten: a retest or a re-grade publishes a new result and keeps the earlier one readable.
Act on this result
Nothing filed here edits the run or the grade. A challenge opens a review, and a review can produce a new run or a re-grade — which becomes the current result and leaves this one in the history.
You run the same kind of test against your own setup and get the same behaviour.
Agree →Yours behaves differently. Tell us what you got, with a screenshot if you have one.
Disagree →Something here is wrong — a reference value, a transcription, a grade.
Report an issue →Tell us what changed and we schedule a rerun of the failing test case. The old result stays as history.
Vendor notice →The same record is available as structured data through the AI Demos MCP server, with the counts, the coverage and every per-test-case reason carried as fields.
Verify the proof files
These files support this result. Open a file to inspect the original evidence.
File fingerprints (SHA-256)
A fingerprint identifies the exact file used for this result.