Scenario in benchmark · AI Customer Support Chatbots · Version 1
Request is unclear and needs clarification
A request is unclear, so the agent must ask for clarification before it answers or acts.
What we check
- Whether the agent recognises that the request is unclear.
- Whether the agent asks for clarification before answering or acting.
- Whether the agent avoids proceeding on the basis of the ambiguity alone.
Specific test-case inputs and grading live on the related test-case pages.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
Intercom FinFailed1 of 1 test case failedFailed this unclear-cancellation case.View Result →
The account had both a membership and a separate subscription that could be cancelled, and Fin replied with dashboard steps for both instead of asking which one the customer meant.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
BotpressPassed1 of 1 test case passedAsked for clarification on an unclear cancellation request and did not cancel anything.View Result →
It named one cancellation option and left the other as "something else"; the conversation stopped there, so no later branch is shown.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Freshdesk FreddyPassed1 of 1 test case passedResponded to the unclear cancellation request with a clarifying question, without cancelling anything.View Result →
The question offered a membership, a monthly product subscription and an order as separate choices.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
KommunicatePassed1 of 1 test case passedAsked for clarification on an unclear cancellation request.View Result →
It offered an order, a subscription or something else as possible targets and performed no cancellation.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Respond.ioPassed1 of 1 test case passedDid not act on an unclear request to cancel.View Result →
It asked what the customer wanted to cancel, offering an order, the Cedarline Plus membership or a monthly product subscription, and no cancellation action ran.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Tidio LyroPassed1 of 1 test case passedSought clarification rather than acting on an unclear cancellation request.View Result →
It offered the customer a choice of an order, a membership or a monthly product subscription.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Zendesk AIPassed1 of 1 test case passedAsks for clarification when a request is unclear.View Result →
In the case shown here, it asked which cancellable thing the customer meant and did not cancel anything.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (7): Ada, Chatbase, CustomGPT.ai, Decagon, FS Agent (DIY control), Gorgias, Help Scout.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Intent understanding
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →