Scenario in benchmark · AI Customer Support Chatbots · Version 1

Information from earlier in the conversation is needed later

A later turn or session requires information established earlier in the conversation.

Capability: Conversation continuity1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the assistant uses information established earlier when answering a later turn.
  2. Whether the assistant continues the conversation from prior context without re-asking for already provided details.
  3. Whether the assistant preserves relevant context across turns or sessions when the later request depends on it.

Specific test-case inputs and grading live on the related test-case pages.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 1
Freshdesk FreddyFailed1 of 1 test case failedDid not keep earlier conversation details available for the follow-up.View Result →

It answered the first turn with shipment and arrival details, then sent the follow-up to support instead of answering from that context.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 5
BotpressPassed1 of 1 test case passedKept earlier conversation context across the two turns.View Result →

The second reply used the first turn's order number to give the delivery date and tracking number, and it did not ask for the number again.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Intercom FinPassed1 of 1 test case passedPassed this scenario.View Result →

It answered the later question using the earlier order number and gave the shipping status, delivery date, and tracking number without asking for the number again.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

KommunicatePassed1 of 1 test case passedHandled the later question using information from the first message.View Result →

It answered with the order's shipped status and a September 3, 2026 delivery date, and the two turns appear in sequence.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Respond.ioPassed1 of 1 test case passedPassed this conversation-continuity scenario.View Result →

The second customer message was answered with the earlier order number, and the reply gave a delivery date, carrier, and tracking number without asking for the number again.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Zendesk AIPassed1 of 1 test case passedUsed an earlier order number to answer a later delivery question without asking for it again.View Result →

It correctly identified UPS as the carrier and reported that the order was shipped and not yet marked delivered.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (8): Ada, Chatbase, CustomGPT.ai, Decagon, FS Agent (DIY control), Gorgias, Help Scout, Tidio Lyro.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Conversation continuity

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

Information from earlier in the conversation is needed later in AI Customer Support Chatbots | AI Demos