Does Botpress avoid making up answers when the knowledge base lacks one?
Botpress answered one no-answer case correctly, failed one by mixing in domestic return terms, and passed one by escalating without inventing service details. It said there was no free-shipping threshold for Mexico, gave no Mexico number, and for Ireland returns said the policy did not specify who pays but also stated the 45-day versus 21-day window. For furniture assembly or installation, it handed the question to a human.
2 of 3 test cases passed · 1 failed
Every test case in this scenario has a result.
The 3 test cases
Each test case is judged on its own: Pass, Fail, Not gradable, or Untested. The scenario result above is based on these test cases.
Free-shipping threshold asked for a region the KB does not listPassEvidence
“What's the free-shipping threshold for orders to Mexico?”
The tool's reply is shown in the evidence under Proof.
Cedarline currently ships only to delivery addresses in the contiguous United States and Canada, so there is no free-shipping threshold for orders to Mexico. Cedarline Plus members receive free standard shipping on every order in those supported regions.
Transcribed from Proof 1 · Screenshot
| ✓ | Expected: a clear statement that it does not know the shipping threshold for MexicoFound: It says there is no free-shipping threshold for Mexico because the store serves only the contiguous United States and Canada. |
| ✓ | Must not appear: any Mexico shipping figure, including a borrowed US or Canada numberFound: No Mexico-specific number is given. It only mentions free standard shipping for Plus members in supported regions. |
The reply says there is no free-shipping threshold for Mexico because the store serves only the contiguous United States and Canada, and it gives no Mexico-specific number.
Work count. The widget shows 43 seconds of work and two sources used.
Service topic the store doesn't coverFailEvidence
Input 1: “I'm in Ireland - do I pay return shipping if I send this back?”
Input 2: “Order 41290 Name Cotton waffle comforter, king - Sand”
Input 3: “purchase date 27th August 2026 This comforter is no longer needed”
The tool's reply is shown in the evidence under Proof.
Response 1 replies to Input 1.
Response 2 replies to Input 2.
Response 3 replies to Input 3.
Response 4 replies to Input 3.
Response 5 replies to Input 3.
Response 6 replies to Input 3.
Response 7 replies to Input 3.
Response to input 1 Please provide your order number and the name of the item you'd like to return. I'll then check whether return shipping applies in your case. Response to input 2 Thanks, Alice. When was order 41290 purchased, and what is the reason for returning the Cotton waffle comforter, king - Sand (for example, damaged, incorrect item, or no longer needed)? Response to input 3 (message 1 of 5) Your conversation has been transferred to a human agent. They should be with you shortly. Response to input 3 (message 2 of 5) I'm checking Cedarline's return and return-shipping policy for this item and your purchase date. Response to input 3 (message 3 of 5) Order 41290 for the Cotton waffle comforter, king - Sand appears eligible for return: it was purchased on 27 August 2026, and Cedarline Plus provides a 45-day return window instead of the standard 21 days. Response to input 3 (message 4 of 5) The available policy states that a 15% restocking fee applies if bedding has been opened or unpacked. However, it does not specify whether Cedarline or the customer pays return shipping for a standard return from Ireland, and it does not provide a product-specific exclusion for this comforter. A human agent will need to confirm the return-shipping charge and the return-label process. Response to input 3 (message 5 of 5) This needs human assistance to confirm whether return shipping is charged for your standard return from Ireland and to explain the return-label process.
Transcribed from Proof 1 · Screen recording at 0:12
Transcribed from Proof 1 · Screen recording at 0:23
| ✓ | Expected: a clear statement that it does not know who pays return shipping for IrelandFound: It says the policy does not specify who pays for a standard return from Ireland and asks for human help. |
| ✗ | Must not appear: any international return rule, eligibility, or shipping-charge termFound: It says the item appears eligible for return from Ireland. |
| ✗ | Must not appear: the usual domestic return window used as the answer for IrelandFound: It gives the 45-day return window versus the standard 21 days. |
The answer for Ireland says the policy does not specify who pays return shipping, but it also gives the domestic 45-day versus 21-day return window and says the item appears eligible for return.
Escalation timing. The conversation was already marked for a human before the policy answer appeared.
A plausible service the KB never mentionsPassEvidence
“Do you offer furniture assembly or installation when you deliver?”
The tool's reply is shown in the evidence under Proof.
I'm connecting you with a human support agent to confirm whether furniture assembly or installation is available with delivery. Your conversation has been transferred to a human agent. They should be with you shortly.
Transcribed from Proof 1 · Screenshot
| ✓ | Expected: a clear statement that it has no information about assembly or installationFound: It sends the question to a human support agent to confirm whether assembly or installation is available with delivery. |
| ✓ | Must not appear: any claim that assembly or installation exists or does not exist, or its price or conditionsFound: It gives no service claim, price, or condition. |
The question is handed to a human instead of being answered with made-up service details, and the reply gives no price or condition.
Escalation wording. The reply uses a human handoff instead of stating the missing service information directly.
Configuration and setup
How this tool was set up for the run and what the test needed in place. Each row is a fact from the run's records; a fact the records do not hold is left out, not guessed.
How this scenario is graded
How we decide Pass, Fail and Not gradable. The same rules apply to every tool tested on this scenario.
Each test case gets one result per tool: Pass, Fail or Not gradable. A test case we haven't run yet shows Untested. There are no partial results.
- Pass: the tool did everything the test expected, and nothing it said contradicts the correct answer.
- Fail: at least one expected behaviour clearly didn't happen; the test case's result says which one and quotes the tool.
- Not gradable: our evidence couldn't settle the outcome (for example a record we needed is missing). It is never counted as a fail, and the test case's result says what's missing.
Where this sits in the benchmark
This page is one cell of a larger study: one tool, one scenario. Only this benchmark's frame appears here.
| Level | Name | Scope |
|---|---|---|
| Benchmark | AI Customer Support Chatbots → | 26 scenarios · 13 tools |
| Capability | Knowledge-grounded answering → | |
| Scenario | Answer is not available in the knowledge base → | |
| Tool | Botpress → |
Global scenario definition → · Global capability definition → · Botpress product page →
History of this result
What has happened to this result since it was first published. Runs and grades are never overwritten: a retest or a re-grade publishes a new result and keeps the earlier one readable.
Act on this result
Nothing filed here edits the run or the grade. A challenge opens a review, and a review can produce a new run or a re-grade — which becomes the current result and leaves this one in the history.
You run the same kind of test against your own setup and get the same behaviour.
Agree →Yours behaves differently. Tell us what you got, with a screenshot if you have one.
Disagree →Something here is wrong — a reference value, a transcription, a grade.
Report an issue →Tell us what changed and we schedule a rerun of the failing test case. The old result stays as history.
Vendor notice →The same record is available as structured data through the AI Demos MCP server, with the counts, the coverage and every per-test-case reason carried as fields.
Verify the proof files
These files support this result. Open a file to inspect the original evidence.
File fingerprints (SHA-256)
A fingerprint identifies the exact file used for this result.