Request requires choosing the correct source or action
A request that requires choosing the correct source of truth or the correct action before replying.
What this scenario means
This scenario tests whether the agent can identify what it should rely on, and whether it should answer, look something up, or act first. It is hard because the right response depends on matching the request to the right source or operation, not just producing a plausible answer. A good agent makes the right choice, stays consistent with that choice, and does not answer from the wrong source.
What we evaluate
- Whether the agent selects the correct source of truth before responding.
- Whether the agent chooses the correct action when the request calls for action rather than retrieval.
- Whether the reply is based on the relevant source and not on an unrelated one.
- Whether the agent avoids mixing information from sources that do not apply.
Capabilities this scenario exercises
A scenario may exercise one or more capabilities.
Intent understanding
Works out what the customer actually wants before answering or acting
Benchmarks that use this scenario
A scenario has global identity and may be reused across benchmarks.
AI Customer Support Chatbots
Automate customer support using an AI chatbot / AI customer support agent — which agents handle real customer support conversations for a business best?