Scenario in benchmark · AI Customer Support Chatbots · Version 1
Requested action cannot be completed
A request asks for an action that the system cannot complete.
What we check
- Whether the agent refuses to claim success when the requested action cannot be carried out.
- Whether the agent explains that the action is unavailable, unsupported, or blocked.
- Whether the agent gives the next step or route instead of acting as if the task succeeded.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
BotpressPassed1 of 1 test case passedRefused the address change request and did not claim it was completed.View Result →
It said the order had already shipped, gave the UPS tracking number and follow-up step, and the order record shows the same shipped state and address-change lock.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Freshdesk FreddyPassed1 of 1 test case passedRefused the requested change and gave a next step when the action could not be completed.View Result →
It said the delivery address for order 41982 could not be changed because the order had already shipped, then offered UPS tracking 1Z999AA10123456784 and follow-up help without claiming the change worked.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Intercom FinPassed1 of 1 test case passedRefused the address change instead of claiming it worked.View Result →
It said the order had already shipped, gave next steps, and the order state shows shipped with address changes disabled.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
KommunicatePassed1 of 1 test case passedDeclined the delivery-address change and offered to connect the customer with a human agent.View Result →
It did not claim that the change had succeeded.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Zendesk AIPassed1 of 1 test case passedRefused the address change request on the shipped order and did not claim success.View Result →
It said the address could no longer be changed, gave shipment as the reason, and offered a human agent as the next step.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (9): Ada, Chatbase, CustomGPT.ai, Decagon, FS Agent (DIY control), Gorgias, Help Scout, Respond.io, Tidio Lyro.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 1 public · 0 withheld
- Capabilities exercised here
- Action execution
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →