Scenario in benchmark · AI Customer Support Chatbots · Version 1
The request uses different wording than the source
A request uses wording different from the source, but the system must still resolve the same underlying fact or passage.
What we check
- Whether the system recognises the same underlying fact or passage despite different wording.
- Whether it relies on meaning rather than an exact phrase match.
- Whether it still resolves the correct source when the request is paraphrased.
- Whether it treats different wording as equivalent when the underlying meaning is the same.
Specific test-case inputs and grading live on the related test-case pages.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
Respond.ioPassed1 of 1 test case passedAnswered the reworded return question with the required 15% refund deduction, ruling out a full refund.View Result →
The customer asked about an opened rug without using the knowledge base’s term “restocking fee”.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (13): Ada, Botpress, Chatbase, CustomGPT.ai, Decagon, Freshdesk Freddy, FS Agent (DIY control), Gorgias, Help Scout, Intercom Fin, Kommunicate, Tidio Lyro, Zendesk AI.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Knowledge-grounded answering
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →