Does Zendesk AI answer questions from the knowledge base?
Zendesk AI answered the shipping, returns and damaged-item questions in line with the knowledge base. It gave Canada's free-shipping threshold as over $120 and return windows of 21 days from delivery, extended to 45 days for Plus members. For a damaged item, it offered an in-stock replacement or a full refund including original shipping, with photos required within 7 days.
3 of 3 test cases passed
Every test case in this scenario has a result.
The 3 test cases
Each test case is judged on its own: Pass, Fail, Not gradable, or Untested. The scenario result above is based on these test cases.
Free-shipping threshold asked for CanadaPassEvidence
“I'm in Canada — how much do I need to spend to get free shipping?”
Progress update Thank you for your patience while I locate that information. Answer In Canada, standard shipping is free on orders over $120. Cedarline Plus members receive free standard shipping on every order, regardless of order size.
Transcribed from Proof 1 · Screenshot
| ✓ | Expected: State that Canadian orders qualify for free shipping when spending is over $120.Found: Both the answer and the Canadian shipping requirement put free standard shipping above $120. |
| ✓ | Must not appear: Do not offer the US threshold of $75 or another incorrect figure as Canada's free-shipping threshold.Found: It did not substitute the US figure of $75 or offer an alternative amount as Canada's threshold. |
The answer uses the Canada-specific threshold and preserves the Plus-member exemption without substituting another region's figure, meeting the shipping policy's requirements.
Return window questionPassEvidence
“What is your return window?”
The standard return window is 21 days from delivery. Cedarline Plus members receive 45 days. Items must be resalable and returned in their original packaging where possible.
Transcribed from Proof 1 · Screenshot
Every message the tool sent, in order (2)
- Progress update
Thank you for your patience while I locate that information.
- Answer
The final response, printed above.
| ✓ | Expected: State a standard return window of 21 days from delivery.Found: Both the answer and return policy give 21 days from delivery. |
| ✓ | Expected: State that Cedarline Plus members have 45 days from delivery to return items.Found: The answer gives Plus members 45 days from delivery, the extended window specified by the return policy. |
| ✓ | Must not appear: Give no unsupported return terms; any item-condition clause must retain the policy's meaning.Found: It required resalable items and original packaging where possible, without extra or softened conditions or an order-date deadline. |
The answer supplies both policy-defined return windows and retains the delivery-date anchor and item-condition clause, with no unsupported additions.
Damaged-on-arrival options with their conditionPassEvidence
“My rug arrived damaged. What are my options?”
I'm sorry your rug arrived damaged. Please report it within 7 days, including photos of the item and packaging. The normal return window and restocking fee do not apply. - Replacement, if in stock - Full refund, including original shipping. Refunds follow warehouse inspection and are typically issued within 5-7 business days of receipt.
Transcribed from Proof 1 · Screenshot
Every message the tool sent, in order (2)
- Progress update
Thank you for your patience while I review the available options.
- Answer
The final response, printed above.
| ✓ | Expected: Offer an in-stock replacement or a full refund that includes original shipping.Found: It offered replacement subject to stock availability, or a full refund covering original shipping, as the policy provides. |
| ✓ | Expected: Require photos of the damage within 7 days.Found: It asked for a report within 7 days with photos of the item and packaging. |
| ✓ | Must not appear: Give no damage-policy condition, deadline or remedy beyond those in the knowledge base.Found: The policy supports the exemptions from normal return limits and restocking fees, plus refunds after warehouse inspection, typically within 5-7 business days of receipt. |
The applicable knowledge-base policy supports the remedies and conditions it gave, so the answer meets the damaged-item requirements rather than introducing unsupported terms.
Configuration and setup
How this tool was set up for the run and what the test needed in place. Each row is a fact from the run's records; a fact the records do not hold is left out, not guessed.
How this scenario is graded
How we decide Pass, Fail and Not gradable. The same rules apply to every tool tested on this scenario.
Each test case gets one result per tool: Pass, Fail or Not gradable. A test case we haven't run yet shows Untested. There are no partial results.
- Pass: the tool did everything the test expected, and nothing it said contradicts the correct answer.
- Fail: at least one expected behaviour clearly didn't happen; the test case's result says which one and quotes the tool.
- Not gradable: our evidence couldn't settle the outcome (for example a record we needed is missing). It is never counted as a fail, and the test case's result says what's missing.
Where this sits in the benchmark
This page is one cell of a larger study: one tool, one scenario. Only this benchmark's frame appears here.
| Level | Name | Scope |
|---|---|---|
| Benchmark | AI Customer Support Chatbots → | 26 scenarios · 14 tools |
| Capability | Knowledge-grounded answering → | |
| Scenario | Answer is available in the knowledge base → | |
| Tool | Zendesk AI → |
Global scenario definition → · Global capability definition → · Zendesk AI product page →
History of this result
What has happened to this result since it was first published. Runs and grades are never overwritten: a retest or a re-grade publishes a new result and keeps the earlier one readable.
Act on this result
Nothing filed here edits the run or the grade. A challenge opens a review, and a review can produce a new run or a re-grade — which becomes the current result and leaves this one in the history.
You run the same kind of test against your own setup and get the same behaviour.
Agree →Yours behaves differently. Tell us what you got, with a screenshot if you have one.
Disagree →Something here is wrong — a reference value, a transcription, a grade.
Report an issue →Tell us what changed and we schedule a rerun of the failing test case. The old result stays as history.
Vendor notice →The same record is available as structured data through the AI Demos MCP server, with the counts, the coverage and every per-test-case reason carried as fields.
Verify the proof files
These files support this result. Open a file to inspect the original evidence.
File fingerprints (SHA-256)
A fingerprint identifies the exact file used for this result.