Scenario in benchmark · AI Customer Support Chatbots · Version 1

Request is unclear and needs clarification

A request is unclear, so the agent must ask for clarification before it answers or acts.

Capability: Intent understanding1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the agent recognises that the request is unclear.
  2. Whether the agent asks for clarification before answering or acting.
  3. Whether the agent avoids proceeding on the basis of the ambiguity alone.

Specific test-case inputs and grading live on the related test-case pages.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 1
Intercom FinFailed1 of 1 test case failedFailed this unclear-cancellation case.View Result →

The account had both a membership and a separate subscription that could be cancelled, and Fin replied with dashboard steps for both instead of asking which one the customer meant.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 6
BotpressPassed1 of 1 test case passedAsked for clarification on an unclear cancellation request and did not cancel anything.View Result →

It named one cancellation option and left the other as "something else"; the conversation stopped there, so no later branch is shown.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Freshdesk FreddyPassed1 of 1 test case passedResponded to the unclear cancellation request with a clarifying question, without cancelling anything.View Result →

The question offered a membership, a monthly product subscription and an order as separate choices.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

KommunicatePassed1 of 1 test case passedAsked for clarification on an unclear cancellation request.View Result →

It offered an order, a subscription or something else as possible targets and performed no cancellation.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Respond.ioPassed1 of 1 test case passedDid not act on an unclear request to cancel.View Result →

It asked what the customer wanted to cancel, offering an order, the Cedarline Plus membership or a monthly product subscription, and no cancellation action ran.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Tidio LyroPassed1 of 1 test case passedSought clarification rather than acting on an unclear cancellation request.View Result →

It offered the customer a choice of an order, a membership or a monthly product subscription.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Zendesk AIPassed1 of 1 test case passedAsks for clarification when a request is unclear.View Result →

In the case shown here, it asked which cancellable thing the customer meant and did not cancel anything.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (7): Ada, Chatbase, CustomGPT.ai, Decagon, FS Agent (DIY control), Gorgias, Help Scout.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Intent understanding

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

Request is unclear and needs clarification in AI Customer Support Chatbots | AI Demos