Can Vectara retrieve from the scoped part of the documents?
Vectara kept retrieval within the requested document subset without returning an empty result. The document containing the answer ranked first in an unrestricted search and was absent from the restricted results.
1 of 1 test case passed
Every test case in this scenario has a result.
The test case
Each test case is judged on its own: Pass, Fail, Not gradable, or Untested. The scenario result above is based on this test case.
Search restricted to the subset that does not hold the answerPassEvidence
Input · Unrestricted comparison search: “Which depot holds the Tolvan A3 loan pool?”
Copied from Output file (text), lines 8–43, 46–77
TC93 — Vectara — R6 Managed RAG v1 Query: Which depot holds the Tolvan A3 loan pool? Corpus: r6-managed-rag-ss-v1 Subset A (holds the answer): region=uk (A05_depot_and_commissioning_note.md) Subset B (scoped to, does not hold the answer): region=nordic (A06_nordic_service_addendum.docx) ==================================================================================================== BASELINE — GET /v2/corpora/{corpus_key}/query?query=...&limit=10 (unscoped, full corpus) ==================================================================================================== --- rank 1 | score=0.881304 | doc=A05_depot_and_commissioning_note.md | region=uk --- The Bristol depot holds the Tolvan A3 loan pool: six calibrated units reserved for Critical cover customers. --- rank 2 | score=0.783684 | doc=A06_nordic_service_addendum.docx | region=nordic --- 2.1 What the depot holds Item Held at Bergen Note Brannoc loan pool Two units Issued against a signed loan note Tolvan C5 spares kit One full kit Pressure housings, o-ring sets, end caps Consumables Rolling stock Anodes, guards, gland kits Deck cable, armoured 300 m drum Cut to length on request Loan pool stock is counted on the first working day of each month and the count is sent to the service desk lead the same day. --- rank 3 | score=0.720327 | doc=A10_tolvan_a3_product_overview.md | region=group --- Tolvan A3 - product overview --- rank 4 | score=0.720327 | doc=A10_tolvan_a3_product_overview.md | region=group --- Tolvan A3 - product overview --- rank 5 | score=0.713669 | doc=A10_tolvan_a3_product_overview.md | region=group --- The Tolvan A3 is a conductivity and temperature sensor built forcontinuous deployment in estuarine and coastal water. --- rank 6 | score=0.705271 | doc=A08_warranty_terms_sales_pack.pdf | region=group --- They apply to every new Tolvan A3 supplied by Fenwake Instruments Ltd, however it was ordered and wherever it is deployed. --- rank 7 | score=0.705271 | doc=A09_warranty_statement_service_pack.pdf | region=group --- They apply to every new Tolvan A3 supplied by Fenwake Instruments Ltd, however it was ordered and wherever it is deployed. --- rank 8 | score=0.703798 | doc=A09_warranty_statement_service_pack.pdf | region=group --- Warranty Terms - Tolvan A3 --- rank 9 | score=0.703798 | doc=A08_warranty_terms_sales_pack.pdf | region=group --- Warranty Terms - Tolvan A3 --- rank 10 | score=0.681923 | doc=A01_field_service_policy_and_charges_uk.pdf | region=uk --- It applies to the Tolvan A3, the Tolvan C5 and the Brannoc telemetry gateway, on every cover level. ==================================================================================================== SCOPED — POST /v2/corpora/{corpus_key}/query (search.metadata_filter: doc.region = 'nordic') ==================================================================================================== --- rank 1 | score=0.952838 | doc=A06_nordic_service_addendum.docx | region=nordic --- 2.1 What the depot holds Item Held at Bergen Note Brannoc loan pool Two units Issued against a signed loan note Tolvan C5 spares kit One full kit Pressure housings, o-ring sets, end caps Consumables Rolling stock Anodes, guards, gland kits Deck cable, armoured 300 m drum Cut to length on request Loan pool stock is counted on the first working day of each month and the count is sent to the service desk lead the same day. --- rank 2 | score=0.737914 | doc=A06_nordic_service_addendum.docx | region=nordic --- Requesting a loan unit 3.1 Who to call Loan requests from customers in the region are met from the Bergen depot. --- rank 3 | score=0.619471 | doc=A06_nordic_service_addendum.docx | region=nordic --- Where a customer needs an instrument the depot does not hold as loan stock, the depot raises a request against central stores and tells the customer the same day what can and cannot be met locally. --- rank 4 | score=0.485684 | doc=A06_nordic_service_addendum.docx | region=nordic --- It covers logistics, customs, mobilisation, the regional depot and how loan stock is requested. --- rank 5 | score=0.475575 | doc=A06_nordic_service_addendum.docx | region=nordic --- Do not promise a loan unit before the depot has confirmed one is free. --- rank 6 | score=0.180107 | doc=A06_nordic_service_addendum.docx | region=nordic --- Regional depot The regional depot is at Bergen, on the Damsgard side of the harbour, and is open 07:00 to 16:00 local time, Monday to Friday. --- rank 7 | score=0.140101 | doc=A06_nordic_service_addendum.docx | region=nordic --- Where it is not, the depot charters through a local agent. --- rank 8 | score=0.132739 | doc=A06_nordic_service_addendum.docx | region=nordic --- On request, the depot provides a Norwegian summary of the findings and the actions taken, prepared by the stores controller and checked by the attending engineer. --- rank 9 | score=0.101233 | doc=A06_nordic_service_addendum.docx | region=nordic --- 3.4 Condition on return A returned loan unit is inspected, cleaned and re-certified before it goes back into the pool. --- rank 10 | score=0.091382 | doc=A06_nordic_service_addendum.docx | region=nordic --- 3.2 What can be lent The Bergen loan pool covers the Brannoc telemetry gateway. ==================================================================================================== A05 (the answering doc, region=uk) present in scoped (region=nordic) results: False All 10 scoped results are region=nordic: True
Query results: full unrestricted and Nordic-only result lists
Open the text file ↗| ✓ | Expected: The document containing the answer must not appear when search is restricted to the subset without it.Found: All ten restricted results came from the Nordic service addendum; the UK depot and commissioning note containing the answer was absent. |
The answer-bearing document had to stay out. Its absence from an on-topic, non-empty subset search, despite ranking first without a restriction, demonstrated that Vectara honoured the scope.
Lower-relevance passages still returned. The Nordic service addendum’s ten passages had scores from 0.9528 down to 0.0914; Vectara returned lower-relevance passages from that document rather than fewer results.
Configuration and setup
How this tool was set up for the run and what the test needed in place. Each row is a fact from the run's records; a fact the records do not hold is left out, not guessed.
How this scenario is graded
How we decide Pass, Fail and Not gradable. The same rules apply to every tool tested on this scenario.
Each test case gets one result per tool: Pass, Fail or Not gradable. A test case we haven't run yet shows Untested. There are no partial results.
- Pass: the tool did everything the test expected, and nothing it said contradicts the correct answer.
- Fail: at least one expected behaviour clearly didn't happen; the test case's result says which one and quotes the tool.
- Not gradable: our evidence couldn't settle the outcome (for example a record we needed is missing). It is never counted as a fail, and the test case's result says what's missing.
Where this sits in the benchmark
This page is one cell of a larger study: one tool, one scenario. Only this benchmark's frame appears here.
| Level | Name | Scope |
|---|---|---|
| Benchmark | Managed RAG → | 15 scenarios · 9 tools |
| Capability | Retrieval → | |
| Scenario | Retrieval is scoped to part of the documents → | |
| Tool | Vectara → |
Global scenario definition → · Global capability definition → · Vectara product page →
History of this result
What has happened to this result since it was first published. Runs and grades are never overwritten: a retest or a re-grade publishes a new result and keeps the earlier one readable.
Act on this result
Nothing filed here edits the run or the grade. A challenge opens a review, and a review can produce a new run or a re-grade — which becomes the current result and leaves this one in the history.
You run the same kind of test against your own setup and get the same behaviour.
Agree →Yours behaves differently. Tell us what you got, with a screenshot if you have one.
Disagree →Something here is wrong — a reference value, a transcription, a grade.
Report an issue →Tell us what changed and we schedule a rerun of the failing test case. The old result stays as history.
Vendor notice →The same record is available as structured data through the AI Demos MCP server, with the counts, the coverage and every per-test-case reason carried as fields.
Verify the proof files
These files support this result. Open a file to inspect the original evidence.
File fingerprints (SHA-256)
A fingerprint identifies the exact file used for this result.