Scenario in benchmark · AI Customer Support Chatbots · Version 1

Request requires choosing the correct source or action

A request requires selecting the correct source or action before answering or acting.

Capability: Intent understanding1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the agent selects the appropriate source before answering.
  2. Whether the agent avoids relying on an irrelevant source.
  3. Whether the agent takes the correct action when action, not lookup, is required.

Specific test-case inputs and grading live on the related test-case pages.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Passed · 7
BotpressPassed1 of 1 test case passedChose the order system for the request and returned the shipped status and tracking number.View Result →

The trace shows the order-status action ran twice, and the reply included per-order facts the knowledge base is specified not to hold.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Freshdesk FreddyPassed1 of 1 test case passedUsed the order system to answer the status request and correctly reported that the order had shipped.View Result →

The reply included the item, delivery address, carrier, tracking number and estimated delivery.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Intercom FinPassed1 of 1 test case passedPassed this order-status request.View Result →

It answered from the order system, and the returned shipped status, carrier, tracking number, and delivery address matched what it told the customer.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

KommunicatePassed1 of 1 test case passedReturned the real status for an order-numbered request.View Result →

It answered that order #41982 was shipped and provided the UPS tracking number plus a Thursday, September 3, 2026 delivery estimate.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Respond.ioPassed1 of 1 test case passedChose the order system rather than the knowledge base for an order-status request.View Result →

Its answer included shipment status, tracking details and an estimated delivery date.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Tidio LyroPassed1 of 1 test case passedUsed the order system, not the knowledge base, and correctly reported the order as shipped.View Result →

Its answer included tracking information and an estimated delivery date.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Zendesk AIPassed1 of 1 test case passedPassed this order-status case.View Result →

The conversation shows one custom action between the customer’s question and the reply, and the reply matches the order system’s shipped status, UPS carrier, tracking number, and estimated delivery date.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (7): Ada, Chatbase, CustomGPT.ai, Decagon, FS Agent (DIY control), Gorgias, Help Scout.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Intent understanding

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

Request requires choosing the correct source or action in AI Customer Support Chatbots | AI Demos