Scenario in benchmark · AI Customer Support Chatbots · Version 1
Update something in an external system
A request that requires updating data in a connected external system.
What we check
- Whether the agent identifies the request as one that needs an external-system update.
- Whether the agent carries out the update when the connected system accepts it.
- Whether the agent reports the outcome of the attempted update clearly and accurately.
- Whether the agent avoids claiming success when the connected system does not apply the change.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
BotpressDifferent test setUpdated something in an external system in this case.View Result →
It changed the order address, the system returned the new stored address, and the reply confirmed the change with that same address.
Published on a different test set, so its counts are not compared here.
Intercom FinDifferent test setChanged the delivery address in the store's order system.View Result →
The order system reported the requested new address as saved, and Fin confirmed the change in chat.
Published on a different test set, so its counts are not compared here.
KommunicateDifferent test setChanged the order’s delivery address in the external store system.View Result →
It confirmed the new address in chat, and the order held the requested address afterwards.
Published on a different test set, so its counts are not compared here.
Respond.ioDifferent test setChanged the delivery address on the customer's order in the order system and confirmed the new Denver address in its reply.View Result →
The order afterwards shows the new address, and the order system logged the change as successful.
Published on a different test set, so its counts are not compared here.
Zendesk AIDifferent test setUpdated the delivery address in the external order system.View Result →
The order showed the old address before the request, the product confirmed the new address, and the same order read back with the new address at the end.
Published on a different test set, so its counts are not compared here.
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (9): Ada, Chatbase, CustomGPT.ai, Decagon, Freshdesk Freddy, FS Agent (DIY control), Gorgias, Help Scout, Tidio Lyro.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Action execution
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 pins a scenario rubric for this scenario. Benchmark methodology →