Scenario in benchmark · AI Customer Support Chatbots · Version 1
Earlier conversation/session needs to be continued
A follow-up depends on resuming an earlier conversation or session from prior context.
What we check
- Whether the assistant uses earlier context to continue the same issue.
- Whether the assistant treats the follow-up as a continuation rather than a fresh request that must be fully re-explained.
- Whether the assistant carries the thread forward across turns or across sessions when the earlier context is available.
Specific test-case inputs and grading live on the related test-case pages.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
BotpressFailed1 of 1 test case failedDid not continue the earlier return conversation when the customer came back.View Result →
After reopening, it asked again for the order number and item, even though session 1 had already established them.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
KommunicateFailed1 of 1 test case failedFails to continue an earlier chat after reopening the widget.View Result →
The restored thread still shows the earlier return request, but the bot asks for more details and then asks for the order number again instead of continuing the thread.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Zendesk AIFailed1 of 1 test case failedDid not continue the earlier return conversation in the follow-up session.View Result →
In the second session, the agent answered generically and then gave support channels instead of naming the earlier order or sofa, while the earlier exchange stayed in the same thread.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Freshdesk FreddyPassed1 of 1 test case passedContinued the return conversation when the customer came back in the same thread.View Result →
It identified the return from the follow-up, accepted “same order,” and then gave the refund timeline without the customer repeating the order number or item.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Intercom FinPassed1 of 1 test case passedContinued the earlier return conversation after the widget was closed and reopened.View Result →
The follow-up reply stayed on return timing and did not ask again for the order number or item.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (9): Ada, Chatbase, CustomGPT.ai, Decagon, FS Agent (DIY control), Gorgias, Help Scout, Respond.io, Tidio Lyro.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Conversation continuity
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →