Scenario in benchmark · AI Customer Support Chatbots · Version 1

Retrieve information from an external system

A request requires information to be retrieved from an external connected system.

Capability: Action execution1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the response uses an external connected system to obtain the needed information.
  2. Whether the answer reflects the information returned by that system.
  3. Whether the agent does not rely on the knowledge base alone when the needed information lives elsewhere.

Specific test-case inputs and grading live on the related test-case pages.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Passed · 5
BotpressPassed1 of 1 test case passedPassed this external-order lookup scenario.View Result →

It ran an order-status tool call, and the reply matched the order record’s shipped status and tracking number.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Freshdesk FreddyPassed1 of 1 test case passedRetrieved information from an external system in this case.View Result →

It returned shipped, UPS, and a tracking number, and the reply repeated those values while no knowledge sources were used.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Intercom FinPassed1 of 1 test case passedRetrieved the shipped order's live status from the external system.View Result →

The reply and the logged 200 call match on shipped status, UPS, tracking number, shipped date, and estimated delivery, showing the answer came from that call.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Respond.ioPassed1 of 1 test case passedRetrieved the order’s status and correctly reported that it had shipped.View Result →

The order-status action ran, and the connected order system also listed shipped.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Zendesk AIPassed1 of 1 test case passedRetrieved information from an external system and reported the returned order status.View Result →

For order 41982, the conversation log shows the Check Order Status action firing successfully and returning shipped, UPS, tracking number 1Z999AA10123456784, and estimated delivery 2026-09-22; the customer reply repeats those same fields. The recording shows the action firing and returned fields, but not the HTTP request line or endpoint.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (9): Ada, Chatbase, CustomGPT.ai, Decagon, FS Agent (DIY control), Gorgias, Help Scout, Kommunicate, Tidio Lyro.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Action execution

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

Retrieve information from an external system in AI Customer Support Chatbots | AI Demos