Does Botpress answer customer questions when the answer is in its knowledge base?
Botpress answered shipping, returns and damaged-item questions with the knowledge-base policy figures and remedies. It gave Canada's free-shipping threshold as over $120 and return periods of 21 days, or 45 for Plus members. For damage, it requested photos, required reporting within 7 days of delivery and described a replacement if available or a full refund including original shipping, subject to review.
3 of 3 test cases passed
Every test case in this scenario has a result.
The 3 test cases
Each test case is judged on its own: Pass, Fail, Not gradable, or Untested. The scenario result above is based on these test cases.
Free-shipping threshold asked for CanadaPassEvidence
“I'm in Canada — how much do I need to spend to get free shipping?”
In Canada, standard shipping is free on orders over $120. Since you're a Cedarline Plus member, you receive free standard shipping on every order, regardless of the order total.
Transcribed from Proof 1 · Screenshot
| ✓ | Expected: State that orders over $120 qualify for free shipping in Canada.Found: It stated that standard shipping in Canada is free on orders over $120, the required Canadian threshold. |
| ✓ | Must not appear: Do not present the US threshold of $75, or another figure, as Canada's free-shipping threshold.Found: It gave no competing Canadian threshold and did not present the US threshold of $75 as Canada's. |
The answer met the Canada-specific requirement without substituting another region's figure. Its separate Plus-member benefit did not introduce a competing Canadian threshold.
No shipping source link. The Canada shipping reply has no knowledge-base source link identifying the article behind the $120 threshold.
Return window questionPassEvidence
“What is your return window?”
I'm checking Cedarline's current return policy so I can give you the accurate return window. Cedarline's standard return window is 21 days. As a Cedarline Plus member, you receive an extended 45-day return window instead. Damaged, defective, or incorrect items are handled separately and must be reported within 7 days of delivery, with photos of the item and packaging.
Transcribed from Proof 1 · Screenshot
| ✓ | Expected: State a standard return window of 21 days from delivery.Found: It stated a 21-day standard return window, the policy's required length. |
| ✓ | Expected: State a return window of 45 days from delivery for Plus members.Found: It stated a 45-day return window for Plus members, the policy's required length. |
| ✓ | Must not appear: Do not add a different return period, an order-date starting point or an unsupported or altered return condition.Found: It added the policy's separate rule for damaged, defective or incorrect items: report within 7 days of delivery with photos, without unsupported return terms. |
The answer supplied both required return periods without adding unsupported policy claims. It omitted their delivery-date starting point rather than giving a conflicting one.
Answer despite search error. The return-window reply appeared after a search-failure notice and still listed two knowledge-base sources.
Damaged-on-arrival options with their conditionPassEvidence
Input 1 · Damaged-item question: “My rug arrived damaged. What are my options?”
Input 2 · Order details: “Order number 41290 Name Cotton waffle comforter, king — Sand”
Input 3 · Delivery date: “delivery date 5th September 2026”
Request for order details
I'm sorry your rug arrived damaged. I can help review replacement or refund options. Please provide your order number and the name or description of the rug.
Follow-up question and photo instructions
Thank you. For order 41290—the Cotton Waffle Comforter, King — Sand—when was the purchase made? Since the item arrived damaged, please also keep photos of the comforter and its packaging available; damaged items must be reported within 7 days of delivery.
Final answer: remedies and conditions
Thank you. Your Cotton Waffle Comforter, King — Sand from order 41290 was delivered on September 5, 2026, and the damage was reported within Cedarline's 7-day reporting period. Because it arrived damaged: The standard return window and restocking fee do not apply. Please provide photos of the comforter and its packaging. After review, Cedarline may offer a replacement if available or a full refund, including the original shipping cost. Your return request is eligible for review. If a refund is approved, it is issued after warehouse inspection—typically within 5–7 business days of receipt. Card refunds generally appear within another 5–7 business days; PayPal refunds take 3–5 business days. Please attach clear photos of the damaged comforter and its packaging so the return request can proceed to review. Once received, Cedarline can assess whether a replacement is available or approve a full refund including the original shipping cost.
Transcribed from Proof 1 · Screen recording at 0:28
Every message the tool sent, in order (3)
- Request for order details
Printed above.
- Follow-up question and photo instructions
Printed above.
- Final answer: remedies and conditions
The final response, printed above.
| ✓ | Expected: After review, Cedarline may offer a replacement if available or a full refund including original shipping.Found: It said Cedarline may offer an available replacement or a full refund including original shipping, after review. |
| ✓ | Expected: A photo of the damage must be provided within 7 days.Found: It required damage to be reported within 7 days of delivery and requested photos of the item and packaging. |
| ✓ | Must not appear: Do not add a condition, deadline or remedy beyond the knowledge-base policy.Found: It stated policy-backed terms: no standard return window or restocking fee; refunds after warehouse inspection, typically within 5–7 business days of receipt; cards another 5–7, PayPal 3–5 business days. |
The knowledge-base policy leaves the remedy to Cedarline's discretion. After receiving order details and a delivery date, Botpress gave the policy's conditional replacement-or-refund options and its photo and reporting requirements.
Configuration and setup
How this tool was set up for the run and what the test needed in place. Each row is a fact from the run's records; a fact the records do not hold is left out, not guessed.
How this scenario is graded
How we decide Pass, Fail and Not gradable. The same rules apply to every tool tested on this scenario.
Each test case gets one result per tool: Pass, Fail or Not gradable. A test case we haven't run yet shows Untested. There are no partial results.
- Pass: the tool did everything the test expected, and nothing it said contradicts the correct answer.
- Fail: at least one expected behaviour clearly didn't happen; the test case's result says which one and quotes the tool.
- Not gradable: our evidence couldn't settle the outcome (for example a record we needed is missing). It is never counted as a fail, and the test case's result says what's missing.
Where this sits in the benchmark
This page is one cell of a larger study: one tool, one scenario. Only this benchmark's frame appears here.
| Level | Name | Scope |
|---|---|---|
| Benchmark | AI Customer Support Chatbots → | 26 scenarios · 14 tools |
| Capability | Knowledge-grounded answering → | |
| Scenario | Answer is available in the knowledge base → | |
| Tool | Botpress → |
Global scenario definition → · Global capability definition → · Botpress product page →
History of this result
What has happened to this result since it was first published. Runs and grades are never overwritten: a retest or a re-grade publishes a new result and keeps the earlier one readable.
Act on this result
Nothing filed here edits the run or the grade. A challenge opens a review, and a review can produce a new run or a re-grade — which becomes the current result and leaves this one in the history.
You run the same kind of test against your own setup and get the same behaviour.
Agree →Yours behaves differently. Tell us what you got, with a screenshot if you have one.
Disagree →Something here is wrong — a reference value, a transcription, a grade.
Report an issue →Tell us what changed and we schedule a rerun of the failing test case. The old result stays as history.
Vendor notice →The same record is available as structured data through the AI Demos MCP server, with the counts, the coverage and every per-test-case reason carried as fields.
Verify the proof files
These files support this result. Open a file to inspect the original evidence.
File fingerprints (SHA-256)
A fingerprint identifies the exact file used for this result.