Retrieval is scoped to part of the documents
When retrieval is limited to a defined subset of documents, it should stay inside that scope.
What this scenario means
This tests whether retrieval honours an explicit boundary instead of searching the whole set. It is hard because a system can look correct by returning an answer from outside the allowed subset, or fail by narrowing the search so much that nothing comes back. A good agent returns only results from within scope and leaves out answer-bearing documents that are out of scope.
What we evaluate
- Whether returned results stay within the allowed subset when search is restricted.
- Whether the answer-bearing document outside the allowed subset is excluded from retrieval.
- Whether the system treats a narrow scope as a true no-result case rather than a successful hit.
- Whether the restriction is respected instead of being silently ignored.
Capabilities this scenario exercises
A scenario may exercise one or more capabilities.
Retrieval
The passage that answers the question comes back when the documents hold it — and the absence is discoverable when they don't. Judged against the documents: did the supporting passage come back?
Benchmarks that use this scenario
A scenario has global identity and may be reused across benchmarks.
Managed RAG
Which managed RAG service turns a folder of private documents into a query API that finds the right passage, answers only from what it retrieved, cites the source that actually supports the claim — and keeps up when the source changes?