Update something in an external system
A request that requires updating data in a connected external system.
What this scenario means
This scenario checks whether the agent can carry out a change in a connected system instead of only describing information. It is hard because the agent must recognise that the task is an update, send the right action to the external system, and then report the result clearly. A good agent completes the change when the system allows it, or explains that it could not be done when the system rejects the update.
What we evaluate
- Whether the agent identifies the request as one that needs an external-system update.
- Whether the agent carries out the update when the connected system accepts it.
- Whether the agent reports the outcome of the attempted update clearly and accurately.
- Whether the agent avoids claiming success when the connected system does not apply the change.
Capabilities this scenario exercises
A scenario may exercise one or more capabilities.
Action execution
Looks things up and makes changes in connected systems (orders, subscriptions, tickets)
Benchmarks that use this scenario
A scenario has global identity and may be reused across benchmarks.
AI Customer Support Chatbots
Automate customer support using an AI chatbot / AI customer support agent — which agents handle real customer support conversations for a business best?