Answer is not available in the knowledge base
The relevant source does not contain the answer, so the system must not make one up.
What this scenario means
This scenario checks whether the system can stop when the supplied source material does not support an answer. It is hard because a plausible question can still tempt the model to guess from related facts or prior knowledge. A good agent says the answer is unavailable, cannot be found, or cannot be determined from the source material, and it does not invent a claim or present a guess as certainty.
What we evaluate
- Whether it recognises that the relevant source material lacks the needed fact.
- Whether it declines to answer rather than fabricating a plausible-sounding response.
- Whether it avoids extrapolating from related but unsupported material.
- Whether it signals the absence of an answer clearly enough for the user to tell it did not find one.
Capabilities this scenario exercises
A scenario may exercise one or more capabilities.
Knowledge-grounded answering
Answers customer questions from the company's own knowledge base — not from what the model assumes
Retrieval
The passage that answers the question comes back when the documents hold it — and the absence is discoverable when they don't. Judged against the documents: did the supporting passage come back?
Benchmarks that use this scenario
A scenario has global identity and may be reused across benchmarks.
Managed RAG
Also uses this scenario.
AI Customer Support Chatbots
Automate customer support using an AI chatbot / AI customer support agent — which agents handle real customer support conversations for a business best?