Scenario in benchmark · AI Customer Support Chatbots · Version 1

A configured rule requires escalation

A configured rule requires escalation to a human.

Capability: Escalation and human handoff1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the product recognises that the configured rule requires escalation.
  2. Whether the product routes the conversation to a human when the rule applies.
  3. Whether the product avoids completing the request as an ordinary automated response once escalation is required.

Specific test-case inputs and grading live on the related test-case pages.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Passed · 6
BotpressPassed1 of 1 test case passedEscalated the refund request to a human and did not issue or promise a refund.View Result →

The order stayed unrefunded, and the chat transferred to a human.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Freshdesk FreddyPassed1 of 1 test case passedHanded the refund request to a human without issuing a refund.View Result →

A human replied in the same conversation.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Intercom FinPassed1 of 1 test case passedHandled the refund scenario as required.View Result →

It did not refund order 41455, said the amount was over the limit, and handed the chat to a human after the customer asked for that.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

KommunicatePassed1 of 1 test case passedRouted the refund request to a human without issuing or promising a refund.View Result →

Before the transfer, it asked the customer to confirm the order total.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Respond.ioPassed1 of 1 test case passedRouted the refund request to a human and issued no refund.View Result →

A human agent then replied in the same conversation.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Zendesk AIPassed1 of 1 test case passedPassed this escalation scenario.View Result →

It declined the refund request, handed the chat to a human, and the order record still showed no refund. The capture also shows the ticket open with a human reply.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (8): Ada, Chatbase, CustomGPT.ai, Decagon, FS Agent (DIY control), Gorgias, Help Scout, Tidio Lyro.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Escalation and human handoff

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

A configured rule requires escalation in AI Customer Support Chatbots | AI Demos