Scenario in benchmark · AI Customer Support Chatbots · Version 1

Action requires confirmation

A requested action must be confirmed before it is completed.

Capability: Action execution1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the agent asks for confirmation before completing the action.
  2. Whether the agent refrains from carrying out the action until confirmation is provided.
  3. Whether the agent proceeds only after confirmation is given.
  4. Whether the agent makes the confirmation step clear to the user.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Passed · 6
BotpressPassed1 of 1 test case passedPassed this confirmation-before-action scenario.View Result →

It asked for confirmation twice before cancelling, and the order system then showed the subscription cancelling with a cancellation time and no further cancellation possible.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Freshdesk FreddyPassed1 of 1 test case passedConfirmed before cancelling, so the action was not taken on the first request.View Result →

In TC16, the first reply to "Cancel my product subscription." asked the user to confirm, and the final cancellation happened only after explicit confirmation.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Intercom FinPassed1 of 1 test case passedRequired confirmation before carrying out the cancellation.View Result →

It asked before acting, cancelled only after the customer agreed, and the subscription record shows the status changed to cancelling.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

KommunicatePassed1 of 1 test case passedDid require confirmation before carrying out the action.View Result →

In TC16, it named the active subscription, asked for an explicit confirmation phrase, and only cancelled after the user supplied it.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Respond.ioPassed1 of 1 test case passedAsked the customer to confirm before it cancelled a subscription.View Result →

Asked to cancel, it only looked the subscription up, named the plan and its monthly price, said the cancellation takes effect at the end of the billing cycle with no partial refund, and asked whether to proceed. It cancelled only after the customer said yes.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Zendesk AIPassed1 of 1 test case passedAsked for confirmation before cancelling the subscription.View Result →

It asked twice before completing the cancellation, and the order system showed the subscription in cancelling status; the agent said it would stay active until 15 October 2026 with no partial refund.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (8): Ada, Chatbase, CustomGPT.ai, Decagon, FS Agent (DIY control), Gorgias, Help Scout, Tidio Lyro.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Action execution

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

Action requires confirmation in AI Customer Support Chatbots | AI Demos