Can FutureSmart Document Intelligence keep retrieval scoped to part of the documents?
FutureSmart Document Intelligence failed to keep retrieval scoped to the subset that did not hold the answer. Both restricted searches still returned the answering depot note first at score 1.0000.
1 of 1 test case failed
Every test case in this scenario has a result.
The test case
Each test case is judged on its own: Pass, Fail, Not gradable, or Untested. The scenario result above counts this row.
Search restricted to the subset that does not hold the answerFailEvidence
Input 1: “Which depot holds the Tolvan A3 loan pool?”
Input 2: “{"query": "Which depot holds the Tolvan A3 loan pool?", "filters": {"document_type": "contract"}}”
Input 3: “{"query": "Which depot holds the Tolvan A3 loan pool?", "filters": {"region": "nordic"}}”
Response 1, the final response, answers Input 2.
The Bristol depot holds the Tolvan A3 loan pool: six calibrated units reserved for Critical cover customers.
Transcribed from Proof 2 · Screenshot
--- Collection document_type metadata (GET /collections/{CID}/documents/{id}/fields) --- GET /collections/{CID}/documents -> 8 documents, all status=indexed A05_depot_and_commissioning_note.md document_id=883f27f0-81c3-4c46-90f8-b71f5d3695b6 document_type=documentation A03_calibration_certificate_scan.pdf document_id=3521148f-e319-4db6-bb5b-b5024db6bc12 document_type=certificate A01_field_service_policy_and_charges_uk.pdf document_id=88399604-1153-40ef-be66-f4da43a873a2 document_type=policy A06_nordic_service_addendum.docx document_id=7ca3a7b8-634b-4fb6-91b1-f4cde198aa86 document_type=contract A09_warranty_statement_service_pack.pdf document_id=024a3b75-1900-4fdc-abe2-cee5137bcd17 document_type=contract A08_warranty_terms_sales_pack.pdf document_id=89619c96-ea19-4a34-94ee-9dc193dcbdc0 document_type=contract A04_employee_handbook.docx document_id=09c99644-e630-44a4-81f5-636ed80d942c document_type=handbook A10_tolvan_a3_product_overview.md document_id=db8039d8-7204-416d-9af1-7c625be5b57c document_type=documentation region metadata: not set on any of the 8 documents (collection was never ingested with region attached at upload time) --- API call (baseline, unrestricted) --- POST /collections/{CID}/search Body: {"query": "Which depot holds the Tolvan A3 loan pool?"} --- Raw retrieved response (baseline, full content) --- status: 200 | sections returned: 3 score=1.0000 document_id=883f27f0-81c3-4c46-90f8-b71f5d3695b6 Extracted Text: # Depot, spares and commissioning - working note Bristol depot and UK commissioning engineers. Not a contractual document. ## The depot The Bristol depot sits behind the workshop on the Avonmouth side and is open 07:00 to 17:00, Monday to Friday. Out of hours, the duty engineer holds a key and may draw stock, recording what was taken on the ticket before leaving the building. ## Loan stock The Bristol depot holds the Tolvan A3 loan pool: six calibrated units reserved for Critical cover customers. A loan unit goes out configured for the customer's deployment, with a current certificate in the case, so that it can be rigged without a bench check. Loan units are issued against a signed loan note naming the ticket, the serial number that went out and the serial number it stands in for. The stores controller counts loan stock on the first working day of each month. A customer on Standard or Extended cover may borrow a unit where one is free, but has no reservation on it: a Critical cover customer takes precedence and the borrowed unit is recalled with two working days' notice. ## Consumable stock Anodes, anti-fouling guards and gland kits are held as rolling stock and reordered at the reorder level marked on the bin. The stores controller reviews reorder levels each quarter against the previous four quarters of consumption. ## Couriers The courier cut-off is 15:30 for next-day delivery within the United Kingdom. Anything handed over after the cut-off travels the following working day. Dangerous goods - which means any lithium cell over the threshold on the shipping matrix - need the packing declaration signed by a trained shipper, and there are two in the building. ## What the depot does not do The depot does not calibrate. An instrument that needs a certificate goes to the laboratory, is booked in under its own job number, and comes back to the depot only once the certificate has been signed. score=0.6667 document_id=7ca3a7b8-634b-4fb6-91b1-f4cde198aa86 File Name: A06_nordic_service_addendum.docx Extracted Text: # **Nordic Service Addendum** *Fenwake Instruments Ltd - Norway, Sweden and Denmark* ## **1. What this addendum covers** This addendum sets out how service work is delivered for customers in Norway, Sweden and Denmark. It covers logistics, customs, mobilisation, the regional depot and how loan stock is requested. It does not restate commercial terms, and it does not describe the goods warranty carried by a new instrument. ## **2. Regional depot** The regional depot is at Bergen, on the Damsgard side of the harbour, and is open 07:00 to 16:00 local time, Monday to Friday. It is staffed by two engineers and a stores controller. ## **2.1 What the depot holds** <table> <tr> <th> <b>Item</b> </th> <th> <b>Held at Bergen</b> </th> <th> <b>Note</b> </th> </tr> <tr> <td> Brannoc loan pool </td> <td> Two units </td> <td> Issued against a signed loan note </td> </tr> <tr> <td> Tolvan C5 spares kit </td> <td> One full kit </td> <td> Pressure housings, o-ring sets, end caps </td> </tr> <tr> <td> Consumables </td> <td> Rolling stock </td> <td> Anodes, guards, gland kits </td> </tr> <tr> <td> Deck cable, armoured </td> <td> 300 m drum </td> <td> Cut to length on request </td> </tr> </table> Loan pool stock is counted on the first working day of each month and the count is sent to the service desk lead the same day. A discrepancy is investigated before the next issue. ## **3. Requesting a loan unit** ### **3.1 Who to call** Loan requests from customers in the region are met from the Bergen depot. Ring the depot rather than raising a stores request in the system: the stores queue is worked from the workshop and adds a day to anything that could have been handed over the counter. ### **3.2 What can be lent** score=0.5000 document_id=7ca3a7b8-634b-4fb6-91b1-f4cde198aa86 ### **3.2 What can be lent** The Bergen loan pool covers the Brannoc telemetry gateway. A gateway goes out with its own SIM and a configuration file matching the customer's deployment. Where a customer needs an instrument the depot does not hold as loan stock, the depot raises a request against central stores and tells the customer the same day what can and cannot be met locally. Do not promise a loan unit before the depot has confirmed one is free. ### **3.3 The loan note** --- Every loan is recorded on a signed loan note carrying the ticket number, the serial number issued and the date it is due back. Ten working days is the standard loan period, counted from the day the customer's own instrument is returned to service. An overdue loan is chased by the stores controller at ten and at twenty working days, and after that by the service desk lead. **3.4 Condition on return** A returned loan unit is inspected, cleaned and re-certified before it goes back into the pool. A unit returned with a broken calibration seal is treated as damaged, quoted for repair, and does not go back into the pool until the laboratory has passed it. **4. Customs and export** Instruments moving between the workshop and a Norwegian site travel under a temporary admission carnet. The carnet number goes on the ticket before the instrument leaves the building; an instrument that travels without one is held at the border and the ticket clock is suspended until it is released. **4.1 Repairs that cross a border** Where a repair cannot be completed at Bergen and the instrument must return to the workshop, the depot raises a return note and the customer is told the same day. Return shipping is arranged by the depot, never by the customer, so that the carnet stays valid. **5. Mobilisation** --- API call (restricted — should exclude A05, the answering doc) --- POST /collections/{CID}/search Body: {"query": "Which depot holds the Tolvan A3 loan pool?", "filters": {"document_type": "contract"}} POST /collections/{CID}/search Body: {"query": "Which depot holds the Tolvan A3 loan pool?", "filters": {"region": "nordic"}} --- Result of both restricted calls --- filters={'document_type': 'contract'} status: 200 | sections returned: 3 | top: score=1.0000 A05 <- NOT excluded filters={'region': 'nordic'} status: 200 | sections returned: 3 | top: score=1.0000 A05 <- NOT excluded The two responses are identical (same 3 sections, same scores, same text) and identical to the baseline. Printed once below. --- Raw retrieved response (both restricted calls, full content for every chunk returned) --- score=1.0000 document_id=883f27f0-81c3-4c46-90f8-b71f5d3695b6 <- A05, NOT excluded Extracted Text: # Depot, spares and commissioning - working note Bristol depot and UK commissioning engineers. Not a contractual document. ## The depot The Bristol depot sits behind the workshop on the Avonmouth side and is open 07:00 to 17:00, Monday to Friday. Out of hours, the duty engineer holds a key and may draw stock, recording what was taken on the ticket before leaving the building. ## Loan stock The Bristol depot holds the Tolvan A3 loan pool: six calibrated units reserved for Critical cover customers. A loan unit goes out configured for the customer's deployment, with a current certificate in the case, so that it can be rigged without a bench check. Loan units are issued against a signed loan note naming the ticket, the serial number that went out and the serial number it stands in for. The stores controller counts loan stock on the first working day of each month. A customer on Standard or Extended cover may borrow a unit where one is free, but has no reservation on it: a Critical cover customer takes precedence and the borrowed unit is recalled with two working days' notice. ## Consumable stock Anodes, anti-fouling guards and gland kits are held as rolling stock and reordered at the reorder level marked on the bin. The stores controller reviews reorder levels each quarter against the previous four quarters of consumption. ## Couriers The courier cut-off is 15:30 for next-day delivery within the United Kingdom. Anything handed over after the cut-off travels the following working day. Dangerous goods - which means any lithium cell over the threshold on the shipping matrix - need the packing declaration signed by a trained shipper, and there are two in the building. ## What the depot does not do The depot does not calibrate. An instrument that needs a certificate goes to the laboratory, is booked in under its own job number, and comes back to the depot only once the certificate has been signed. score=0.6667 document_id=7ca3a7b8-634b-4fb6-91b1-f4cde198aa86 File Name: A06_nordic_service_addendum.docx Extracted Text: # **Nordic Service Addendum** *Fenwake Instruments Ltd - Norway, Sweden and Denmark* ## **1. What this addendum covers** This addendum sets out how service work is delivered for customers in Norway, Sweden and Denmark. It covers logistics, customs, mobilisation, the regional depot and how loan stock is requested. It does not restate commercial terms, and it does not describe the goods warranty carried by a new instrument. ## **2. Regional depot** The regional depot is at Bergen, on the Damsgard side of the harbour, and is open 07:00 to 16:00 local time, Monday to Friday. It is staffed by two engineers and a stores controller. ## **2.1 What the depot holds** <table> <tr> <th> <b>Item</b> </th> <th> <b>Held at Bergen</b> </th> <th> <b>Note</b> </th> </tr> <tr> <td> Brannoc loan pool </td> <td> Two units </td> <td> Issued against a signed loan note </td> </tr> <tr> <td> Tolvan C5 spares kit </td> <td> One full kit </td> <td> Pressure housings, o-ring sets, end caps </td> </tr> <tr> <td> Consumables </td> <td> Rolling stock </td> <td> Anodes, guards, gland kits </td> </tr> <tr> <td> Deck cable, armoured </td> <td> 300 m drum </td> <td> Cut to length on request </td> </tr> </table> Loan pool stock is counted on the first working day of each month and the count is sent to the service desk lead the same day. A discrepancy is investigated before the next issue. ## **3. Requesting a loan unit** ### **3.1 Who to call** Loan requests from customers in the region are met from the Bergen depot. Ring the depot rather than raising a stores request in the system: the stores queue is worked from the workshop and adds a day to anything that could have been handed over the counter. ### **3.2 What can be lent** score=0.5000 document_id=7ca3a7b8-634b-4fb6-91b1-f4cde198aa86 ### **3.2 What can be lent** The Bergen loan pool covers the Brannoc telemetry gateway. A gateway goes out with its own SIM and a configuration file matching the customer's deployment. Where a customer needs an instrument the depot does not hold as loan stock, the depot raises a request against central stores and tells the customer the same day what can and cannot be met locally. Do not promise a loan unit before the depot has confirmed one is free. ### **3.3 The loan note** --- Every loan is recorded on a signed loan note carrying the ticket number, the serial number issued and the date it is due back. Ten working days is the standard loan period, counted from the day the customer's own instrument is returned to service. An overdue loan is chased by the stores controller at ten and at twenty working days, and after that by the service desk lead. **3.4 Condition on return** A returned loan unit is inspected, cleaned and re-certified before it goes back into the pool. A unit returned with a broken calibration seal is treated as damaged, quoted for repair, and does not go back into the pool until the laboratory has passed it. **4. Customs and export** Instruments moving between the workshop and a Norwegian site travel under a temporary admission carnet. The carnet number goes on the ticket before the instrument leaves the building; an instrument that travels without one is held at the border and the ticket clock is suspended until it is released. **4.1 Repairs that cross a border** Where a repair cannot be completed at Bergen and the instrument must return to the workshop, the depot raises a return note and the customer is told the same day. Return shipping is arranged by the depot, never by the customer, so that the carnet stays valid. **5. Mobilisation**
| ✗ | Expected: The answering document must not be returned.Found: Both restricted searches still returned the answering document first at score 1.0000. |
The search was supposed to leave out the document that answers the question. Both restricted calls still returned that document first at score 1.0000, so the answer stayed in view instead of being excluded.
Absent field still matched. A filter on the region field still returned the same three results as the unrestricted search.
Three sections returned. The search returned three sections, even though the documented default is five.
The search returned passages rather than a generated answer.
One question was tested with two filters: document type and region.
Configuration and setup
How this tool was set up for the run and what the test needed in place. Each row is a fact from the run's records; a fact the records do not hold is left out, not guessed.
How this scenario is graded
How we decide Pass, Fail and Not gradable. The same rules apply to every tool tested on this scenario.
Each test case gets one result per tool: Pass, Fail or Not gradable. A test case we haven't run yet shows Untested. There are no partial results.
- Pass: the tool did everything the test expected, and nothing it said contradicts the correct answer.
- Fail: at least one expected behaviour clearly didn't happen; the row says which and quotes the tool.
- Not gradable: our evidence couldn't settle the outcome (for example a record we needed is missing). It is never counted as a fail, and the row says what's missing.
Where this sits in the benchmark
This page is one cell of a larger study: one tool, one scenario. Only this benchmark's frame appears here.
| Level | Name | Scope |
|---|---|---|
| Benchmark | Managed RAG → | 15 scenarios · 8 tools |
| Capability | Retrieval → | |
| Scenario | Retrieval is scoped to part of the documents → | |
| Tool | FutureSmart Document Intelligence → |
Global scenario definition → · Global capability definition →
History of this result
What has happened to this result since it was first published. Runs and grades are never overwritten: a retest or a re-grade publishes a new result and keeps the earlier one readable.
Act on this result
Nothing filed here edits the run or the grade. A challenge opens a review, and a review can produce a new run or a re-grade — which becomes the current result and leaves this one in the history.
You run the same kind of test against your own setup and get the same behaviour.
Agree →Yours behaves differently. Tell us what you got, with a screenshot if you have one.
Disagree →Something here is wrong — a reference value, a transcription, a grade.
Report an issue →Tell us what changed and we schedule a rerun of the failing test case. The old result stays as history.
Vendor notice →The same record is available as structured data through the AI Demos MCP server, with the counts, the coverage and every per-test-case reason carried as fields.
Verify the proof files
These files support this result. Open a file to inspect the original evidence.
File fingerprints (SHA-256)
A fingerprint identifies the exact file used for this result.