Scenario definition · S15
Requested action cannot be completed
A request asks for an action that the system cannot complete.
What this scenario means
This tests whether the agent recognises a hard limit in the connected workflow and does not present a failed action as done. A good agent refuses completion, explains the limit clearly, and points to the next step or handoff path where one exists.
What we evaluate
- Whether the agent refuses to claim success when the requested action cannot be carried out.
- Whether the agent explains that the action is unavailable, unsupported, or blocked.
- Whether the agent gives the next step or route instead of acting as if the task succeeded.
◌Test-case detail. Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
Capabilities this scenario exercises
A scenario may exercise one or more capabilities.
Action execution
Looks things up and makes changes in connected systems (orders, subscriptions, tickets)
Benchmarks that use this scenario
A scenario has global identity and may be reused across benchmarks.
AI Customer Support Chatbots
Automate customer support using an AI chatbot / AI customer support agent — which agents handle real customer support conversations for a business best?