Scenario in benchmark · AI Customer Support Chatbots · Version 1

The request uses different wording than the source

A request uses wording different from the source, but the system must still resolve the same underlying fact or passage.

Capability: Knowledge-grounded answering1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the system recognises the same underlying fact or passage despite different wording.
  2. Whether it relies on meaning rather than an exact phrase match.
  3. Whether it still resolves the correct source when the request is paraphrased.
  4. Whether it treats different wording as equivalent when the underlying meaning is the same.

Specific test-case inputs and grading live on the related test-case pages.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Passed · 1
Respond.ioPassed1 of 1 test case passedAnswered the reworded return question with the required 15% refund deduction, ruling out a full refund.View Result →

The customer asked about an opened rug without using the knowledge base’s term “restocking fee”.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (13): Ada, Botpress, Chatbase, CustomGPT.ai, Decagon, Freshdesk Freddy, FS Agent (DIY control), Gorgias, Help Scout, Intercom Fin, Kommunicate, Tidio Lyro, Zendesk AI.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Knowledge-grounded answering

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

The request uses different wording than the source in AI Customer Support Chatbots | AI Demos