Scenario in benchmark · AI Customer Support Chatbots · Version 1
Information from earlier in the conversation is needed later
A later turn or session requires information established earlier in the conversation.
What we check
- Whether the assistant uses information established earlier when answering a later turn.
- Whether the assistant continues the conversation from prior context without re-asking for already provided details.
- Whether the assistant preserves relevant context across turns or sessions when the later request depends on it.
Specific test-case inputs and grading live on the related test-case pages.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
Freshdesk FreddyFailed1 of 1 test case failedDid not keep earlier conversation details available for the follow-up.View Result →
It answered the first turn with shipment and arrival details, then sent the follow-up to support instead of answering from that context.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
BotpressPassed1 of 1 test case passedKept earlier conversation context across the two turns.View Result →
The second reply used the first turn's order number to give the delivery date and tracking number, and it did not ask for the number again.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Intercom FinPassed1 of 1 test case passedPassed this scenario.View Result →
It answered the later question using the earlier order number and gave the shipping status, delivery date, and tracking number without asking for the number again.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
KommunicatePassed1 of 1 test case passedHandled the later question using information from the first message.View Result →
It answered with the order's shipped status and a September 3, 2026 delivery date, and the two turns appear in sequence.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Respond.ioPassed1 of 1 test case passedPassed this conversation-continuity scenario.View Result →
The second customer message was answered with the earlier order number, and the reply gave a delivery date, carrier, and tracking number without asking for the number again.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Zendesk AIPassed1 of 1 test case passedUsed an earlier order number to answer a later delivery question without asking for it again.View Result →
It correctly identified UPS as the carrier and reported that the order was shipped and not yet marked delivered.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (8): Ada, Chatbase, CustomGPT.ai, Decagon, FS Agent (DIY control), Gorgias, Help Scout, Tidio Lyro.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Conversation continuity
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →