Scenario definition · S13
Retrieve information from an external system
A request can be answered only by querying a connected external system.
What this scenario means
This scenario checks whether the agent uses a connected system as the source of truth when it needs current operational information. A good agent looks up the data and reports the returned result accurately, instead of guessing or relying on the knowledge base alone.
What we evaluate
- Whether the agent consults a connected external system when the information is not available from the knowledge base.
- Whether the response reflects the actual data returned by that system.
- Whether the agent avoids inventing, inferring, or substituting an unsupported answer when a lookup is required.
- Whether the answer matches the external system’s current state.
◌Test-case detail. Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
Capabilities this scenario exercises
A scenario may exercise one or more capabilities.
Action execution
Looks things up and makes changes in connected systems (orders, subscriptions, tickets)
Benchmarks that use this scenario
A scenario has global identity and may be reused across benchmarks.
AI Customer Support Chatbots
Automate customer support using an AI chatbot / AI customer support agent — which agents handle real customer support conversations for a business best?