Botpress in AI Customer Support Chatbots
Scenario-level performance from current published Results.
19 scenarios with published Results · 26 scenarios in the benchmark
How Botpress performed
Open a capability to explore its scenarios. Each row reports the test set in its published Result; counts are not combined into an overall score.
Action execution5 scenarios · 3 with published Results
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Action requires confirmation | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBotpress passed this confirmation-before-action scenario. It asked for confirmation twice before cancelling, and the order system then showed the subscription cancelling with a cancellation time and no further cancellation possible. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Retrieve information from an external system | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBotpress passed this external-order lookup scenario. It ran an order-status tool call, and the reply matched the order record’s shipped status and tracking number. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| User is not authorized to perform the action | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBotpress refused an action the user was not authorized to perform. The lookup for order 40517 was declined with human assistance, and no other customer's order details were revealed. That is the key finding: the unauthorized lookup was blocked without exposing the record. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Requested action cannot be completed | No published result | ||
| Update something in an external system | No published result | ||
Agent configuration and control2 scenarios · 2 with published Results
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Configured instruction changes agent behavior | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBotpress followed the configured instruction in the tested chat. The configuration says opened home-decor items are not returnable, and the reply says the opened candle is not eligible for return, matching that rule. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Configured restriction is respected | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBotpress respected the configured restriction in this case. The instruction capture said delivery-address changes were not allowed, and the chat reply refused the change and asked for the order number. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Analytics and observability3 scenarios · 3 with published Results
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Agent actions/tool calls can be inspected | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBotpress lets a buyer inspect an order-lookup tool call in Monitor. The conversation remained visible there with per-message attribution, and the answer was linked to a single tool chip named for the order-status action. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Conversation activity can be inspected | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBotpress lets a user inspect conversation activity in the tested case. The activity list showed 42 conversations with date, duration, message and status columns, and two earlier shipped-order conversations were found and opened with their transcripts visible. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Reported analytics match what actually happened | 0 Pass1 Fail0 Not gradable 1 of 1 test case failedWhat happenedBotpress did not show analytics that matched what actually happened in this run. In the 90-day overview, it reported 0 escalated tickets, while the same window also showed three human handoffs, six human-only resolutions, and a volume breakdown of 37 resolved tickets with 6 human-only. The window covered the whole workspace. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Conversation continuity3 scenarios · 3 with published Results
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Customer corrects information given earlier | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBotpress handled a customer correcting information given earlier in the visible chat. The agent first answered about one order, then switched to the corrected order and said it was processing with no tracking number. The first order's shipped status and tracking number did not carry over. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Earlier conversation/session needs to be continued | 0 Pass1 Fail0 Not gradable 1 of 1 test case failedWhat happenedBotpress failed to continue an earlier conversation when the customer came back later. The recording shows one return conversation ending with a summary and human handoff, then the widget reopening empty. In the later exchange, Botpress asked again for the order number and item instead of picking up where the earlier conversation left off. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Information from earlier in the conversation is needed later | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBotpress used earlier conversation information in the later reply. In the two-turn capture, the first message gives order 41982 and the second answer uses that order to provide delivery details without asking for it again. The chat then transfers to a human; that extra step is outside this result. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Escalation and human handoff3 scenarios · 2 with published Results
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| A configured rule requires escalation | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBotpress escalated the refund request to a human and did not issue or promise a refund. The order stayed unrefunded, and the chat transferred to a human. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| User explicitly asks for a human | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBotpress handled the human request successfully in this scenario. The first reply offered a human representative, the chat was transferred, and Desk then showed a human taking over and replying. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Agent cannot resolve the request | No published result | ||
Feedback and learning2 scenarios · 1 with published Results
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Feedback can be submitted | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBotpress took a submitted correction in this scenario. The correction was entered and submitted, and the returned draft fix said it would omit the AI disclosure and keep the response focused; the capture stops before any applied answer is shown. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Correction changes future behavior where the product claims learning | No published result | ||
Intent understanding3 scenarios · 3 with published Results
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| One message contains multiple requests | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBotpress answered both parts of a multi-request message in one reply. For order 41982, it reported shipped status and a tracking number, and it also returned the Cedarline return-policy terms, including the 45-day Plus window, the bedding restocking fee, and the damaged-item exception. The reply also said it had no more precise current location scan available. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Request is unclear and needs clarification | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBotpress asked for clarification on an unclear cancellation request and did not cancel anything. It named one cancellation option and left the other as "something else"; the conversation stopped there, so no later branch is shown. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Request requires choosing the correct source or action | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBotpress chose the order system for the request and returned the shipped status and tracking number. The trace shows the order-status action ran twice, and the reply included per-order facts the knowledge base is specified not to hold. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Multilingual support2 scenarios · 2 with published Results
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Customer communicates in another supported language | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBotpress answered the French return-policy question in French and returned the Cedarline policy answer. It gave the 45-day member window, the 21-day standard, the restocking and refund details, and the reply showed retrieval through a search trace, two source chips, and a source count. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Customer switches languages during the conversation | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBotpress kept the conversation context when the customer switched from French to English. The same thread shows a French return-policy answer first, then an English follow-up that repeats the 45-day Plus window and the 21-day standard window. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Knowledge-grounded answering3 scenarios · 0 with published Results
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Answer is available in the knowledge base | No published result | ||
| Answer is not available in the knowledge base | No published result | ||
| The request uses different wording than the source | No published result | ||
Reading these Results
Published evidence and test coverage answer different questions.
Which scenarios have a Result?
A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.
What does each Result cover?
Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.
Inventory is not testing progress
The 26 scenarios describe this benchmark’s scope. They are not an assumed applicability or test-coverage denominator for Botpress.