AI Demos Research — the structured-intelligence platform. Every verdict on these pages opens to the execution behind it.
Graded 23 October 2026

Can FutureSmart Document Intelligence keep retrieval scoped to part of the documents?

FutureSmart Document Intelligence failed to keep retrieval scoped to the subset that did not hold the answer. Both restricted searches still returned the answering depot note first at score 1.0000.

1 of 1 test case failed

Every test case in this scenario has a result.

Pass rate0%0 of 1 gradable
Coverage1 of 1test case with a result
0PassNo test case passed.
1FailMissed at least one thing it was expected to do; the reason and proof are on the row.
0Not gradableEvery result here could be graded.
0UntestedEvery test case in this scenario has a result.
The pass rate is a summary. The evidence is the test case below: what we sent, what we checked, what the tool returned, and the proof.

The test case

Each test case is judged on its own: Pass, Fail, Not gradable, or Untested. The scenario result above counts this row.

Search restricted to the subset that does not hold the answerFailEvidence
What we sent

Input 1: “Which depot holds the Tolvan A3 loan pool?”

Input 2: “{"query": "Which depot holds the Tolvan A3 loan pool?", "filters": {"document_type": "contract"}}”

Input 3: “{"query": "Which depot holds the Tolvan A3 loan pool?", "filters": {"region": "nordic"}}”

What the tool returned

Response 1, the final response, answers Input 2.

The Bristol depot holds the Tolvan A3 loan pool: six calibrated units reserved for Critical cover customers.

Transcribed from Proof 2 · Screenshot

Output file (text)
Text log: document types and both search runs
Text398 lines
--- Collection document_type metadata (GET /collections/{CID}/documents/{id}/fields) ---

GET /collections/{CID}/documents  ->  8 documents, all status=indexed

A05_depot_and_commissioning_note.md           document_id=883f27f0-81c3-4c46-90f8-b71f5d3695b6  document_type=documentation
A03_calibration_certificate_scan.pdf          document_id=3521148f-e319-4db6-bb5b-b5024db6bc12  document_type=certificate
A01_field_service_policy_and_charges_uk.pdf   document_id=88399604-1153-40ef-be66-f4da43a873a2  document_type=policy
A06_nordic_service_addendum.docx              document_id=7ca3a7b8-634b-4fb6-91b1-f4cde198aa86  document_type=contract
A09_warranty_statement_service_pack.pdf       document_id=024a3b75-1900-4fdc-abe2-cee5137bcd17  document_type=contract
A08_warranty_terms_sales_pack.pdf             document_id=89619c96-ea19-4a34-94ee-9dc193dcbdc0  document_type=contract
A04_employee_handbook.docx                    document_id=09c99644-e630-44a4-81f5-636ed80d942c  document_type=handbook
A10_tolvan_a3_product_overview.md             document_id=db8039d8-7204-416d-9af1-7c625be5b57c  document_type=documentation

region metadata: not set on any of the 8 documents (collection was never ingested with region attached at upload time)

--- API call (baseline, unrestricted) ---
POST /collections/{CID}/search
Body: {"query": "Which depot holds the Tolvan A3 loan pool?"}

--- Raw retrieved response (baseline, full content) ---
status: 200 | sections returned: 3

score=1.0000  document_id=883f27f0-81c3-4c46-90f8-b71f5d3695b6
Extracted Text: # Depot, spares and commissioning - working note

Bristol depot and UK commissioning engineers. Not a contractual document.

## The depot

The Bristol depot sits behind the workshop on the Avonmouth side and is open
07:00 to 17:00, Monday to Friday. Out of hours, the duty engineer holds a key
and may draw stock, recording what was taken on the ticket before leaving the
building.

## Loan stock

The Bristol depot holds the Tolvan A3 loan pool: six calibrated units reserved for Critical cover
customers. A loan unit goes out configured for the customer's
deployment, with a current certificate in the case, so that it can be rigged
without a bench check.

Loan units are issued against a signed loan note naming the ticket, the serial
number that went out and the serial number it stands in for. The stores
controller counts loan stock on the first working day of each month.

A customer on Standard or Extended cover may borrow a unit where one is free,
but has no reservation on it: a Critical cover customer takes precedence and the
borrowed unit is recalled with two working days' notice.

## Consumable stock

Anodes, anti-fouling guards and gland kits are held as rolling stock and
reordered at the reorder level marked on the bin. The stores controller reviews
reorder levels each quarter against the previous four quarters of consumption.

## Couriers

The courier cut-off is 15:30 for next-day delivery within the United Kingdom.
Anything handed over after the cut-off travels the following working day.
Dangerous goods - which means any lithium cell over the threshold on the
shipping matrix - need the packing declaration signed by a trained shipper, and
there are two in the building.

## What the depot does not do

The depot does not calibrate. An instrument that needs a certificate goes to the
laboratory, is booked in under its own job number, and comes back to the depot
only once the certificate has been signed.

score=0.6667  document_id=7ca3a7b8-634b-4fb6-91b1-f4cde198aa86
File Name: A06_nordic_service_addendum.docx

Extracted Text:

# **Nordic Service Addendum**

*Fenwake Instruments Ltd - Norway, Sweden and Denmark*

## **1. What this addendum covers**

This addendum sets out how service work is delivered for customers in Norway, Sweden and Denmark. It
covers logistics, customs, mobilisation, the regional depot and how loan stock is requested. It does
not restate commercial terms, and it does not describe the goods warranty carried by a new
instrument.

## **2. Regional depot**

The regional depot is at Bergen, on the Damsgard side of the harbour, and is open 07:00 to 16:00
local time, Monday to Friday. It is staffed by two engineers and a stores controller.

## **2.1 What the depot holds**

<table>
  <tr>
    <th>
      <b>Item</b>
    </th>
    <th>
      <b>Held at Bergen</b>
    </th>
    <th>
      <b>Note</b>
    </th>
  </tr>
<tr>
    <td>
      Brannoc loan pool
    </td>
<td>
      Two units
    </td>
<td>
      Issued against a signed loan note
    </td>
  </tr>
<tr>
    <td>
      Tolvan C5 spares kit
    </td>
<td>
      One full kit
    </td>
<td>
      Pressure housings, o-ring sets, end caps
    </td>
  </tr>
<tr>
    <td>
      Consumables
    </td>
<td>
      Rolling stock
    </td>
<td>
      Anodes, guards, gland kits
    </td>
  </tr>
<tr>
    <td>
      Deck cable, armoured
    </td>
<td>
      300 m drum
    </td>
<td>
      Cut to length on request
    </td>
  </tr>
</table>

Loan pool stock is counted on the first working day of each month and the count is sent to the
service desk lead the same day. A discrepancy is investigated before the next issue.

## **3. Requesting a loan unit**

### **3.1 Who to call**

Loan requests from customers in the region are met from the Bergen depot. Ring the depot rather than
raising a stores request in the system: the stores queue is worked from the workshop and adds a day
to anything that could have been handed over the counter.

### **3.2 What can be lent**

score=0.5000  document_id=7ca3a7b8-634b-4fb6-91b1-f4cde198aa86
### **3.2 What can be lent**

The Bergen loan pool covers the Brannoc telemetry gateway. A gateway goes out with its own SIM and a
configuration file matching the customer's deployment. Where a customer needs an instrument the
depot does not hold as loan stock, the depot raises a request against central stores and tells the
customer the same day what can and cannot be met locally. Do not promise a loan unit before the
depot has confirmed one is free.

### **3.3 The loan note**

---



Every loan is recorded on a signed loan note carrying the ticket number, the serial number issued
and the date it is due back. Ten working days is the standard loan period, counted from the day the
customer's own instrument is returned to service. An overdue loan is chased by the stores controller
at ten and at twenty working days, and after that by the service desk lead.

**3.4 Condition on return**

A returned loan unit is inspected, cleaned and re-certified before it goes back into the pool. A
unit returned with a broken calibration seal is treated as damaged, quoted for repair, and does not
go back into the pool until the laboratory has passed it.

**4. Customs and export**

Instruments moving between the workshop and a Norwegian site travel under a temporary admission
carnet. The carnet number goes on the ticket before the instrument leaves the building; an
instrument that travels without one is held at the border and the ticket clock is suspended until it
is released.

**4.1 Repairs that cross a border**

Where a repair cannot be completed at Bergen and the instrument must return to the workshop, the
depot raises a return note and the customer is told the same day. Return shipping is arranged by the
depot, never by the customer, so that the carnet stays valid.

**5. Mobilisation**

--- API call (restricted — should exclude A05, the answering doc) ---
POST /collections/{CID}/search
Body: {"query": "Which depot holds the Tolvan A3 loan pool?", "filters": {"document_type": "contract"}}
POST /collections/{CID}/search
Body: {"query": "Which depot holds the Tolvan A3 loan pool?", "filters": {"region": "nordic"}}

--- Result of both restricted calls ---
filters={'document_type': 'contract'}  status: 200 | sections returned: 3 | top: score=1.0000  A05  <- NOT excluded
filters={'region': 'nordic'}           status: 200 | sections returned: 3 | top: score=1.0000  A05  <- NOT excluded
The two responses are identical (same 3 sections, same scores, same text) and identical to the baseline.
Printed once below.

--- Raw retrieved response (both restricted calls, full content for every chunk returned) ---
score=1.0000  document_id=883f27f0-81c3-4c46-90f8-b71f5d3695b6  <- A05, NOT excluded
Extracted Text: # Depot, spares and commissioning - working note

Bristol depot and UK commissioning engineers. Not a contractual document.

## The depot

The Bristol depot sits behind the workshop on the Avonmouth side and is open
07:00 to 17:00, Monday to Friday. Out of hours, the duty engineer holds a key
and may draw stock, recording what was taken on the ticket before leaving the
building.

## Loan stock

The Bristol depot holds the Tolvan A3 loan pool: six calibrated units reserved for Critical cover
customers. A loan unit goes out configured for the customer's
deployment, with a current certificate in the case, so that it can be rigged
without a bench check.

Loan units are issued against a signed loan note naming the ticket, the serial
number that went out and the serial number it stands in for. The stores
controller counts loan stock on the first working day of each month.

A customer on Standard or Extended cover may borrow a unit where one is free,
but has no reservation on it: a Critical cover customer takes precedence and the
borrowed unit is recalled with two working days' notice.

## Consumable stock

Anodes, anti-fouling guards and gland kits are held as rolling stock and
reordered at the reorder level marked on the bin. The stores controller reviews
reorder levels each quarter against the previous four quarters of consumption.

## Couriers

The courier cut-off is 15:30 for next-day delivery within the United Kingdom.
Anything handed over after the cut-off travels the following working day.
Dangerous goods - which means any lithium cell over the threshold on the
shipping matrix - need the packing declaration signed by a trained shipper, and
there are two in the building.

## What the depot does not do

The depot does not calibrate. An instrument that needs a certificate goes to the
laboratory, is booked in under its own job number, and comes back to the depot
only once the certificate has been signed.

score=0.6667  document_id=7ca3a7b8-634b-4fb6-91b1-f4cde198aa86
File Name: A06_nordic_service_addendum.docx

Extracted Text:

# **Nordic Service Addendum**

*Fenwake Instruments Ltd - Norway, Sweden and Denmark*

## **1. What this addendum covers**

This addendum sets out how service work is delivered for customers in Norway, Sweden and Denmark. It
covers logistics, customs, mobilisation, the regional depot and how loan stock is requested. It does
not restate commercial terms, and it does not describe the goods warranty carried by a new
instrument.

## **2. Regional depot**

The regional depot is at Bergen, on the Damsgard side of the harbour, and is open 07:00 to 16:00
local time, Monday to Friday. It is staffed by two engineers and a stores controller.

## **2.1 What the depot holds**

<table>
  <tr>
    <th>
      <b>Item</b>
    </th>
    <th>
      <b>Held at Bergen</b>
    </th>
    <th>
      <b>Note</b>
    </th>
  </tr>
<tr>
    <td>
      Brannoc loan pool
    </td>
<td>
      Two units
    </td>
<td>
      Issued against a signed loan note
    </td>
  </tr>
<tr>
    <td>
      Tolvan C5 spares kit
    </td>
<td>
      One full kit
    </td>
<td>
      Pressure housings, o-ring sets, end caps
    </td>
  </tr>
<tr>
    <td>
      Consumables
    </td>
<td>
      Rolling stock
    </td>
<td>
      Anodes, guards, gland kits
    </td>
  </tr>
<tr>
    <td>
      Deck cable, armoured
    </td>
<td>
      300 m drum
    </td>
<td>
      Cut to length on request
    </td>
  </tr>
</table>

Loan pool stock is counted on the first working day of each month and the count is sent to the
service desk lead the same day. A discrepancy is investigated before the next issue.

## **3. Requesting a loan unit**

### **3.1 Who to call**

Loan requests from customers in the region are met from the Bergen depot. Ring the depot rather than
raising a stores request in the system: the stores queue is worked from the workshop and adds a day
to anything that could have been handed over the counter.

### **3.2 What can be lent**

score=0.5000  document_id=7ca3a7b8-634b-4fb6-91b1-f4cde198aa86
### **3.2 What can be lent**

The Bergen loan pool covers the Brannoc telemetry gateway. A gateway goes out with its own SIM and a
configuration file matching the customer's deployment. Where a customer needs an instrument the
depot does not hold as loan stock, the depot raises a request against central stores and tells the
customer the same day what can and cannot be met locally. Do not promise a loan unit before the
depot has confirmed one is free.

### **3.3 The loan note**

---



Every loan is recorded on a signed loan note carrying the ticket number, the serial number issued
and the date it is due back. Ten working days is the standard loan period, counted from the day the
customer's own instrument is returned to service. An overdue loan is chased by the stores controller
at ten and at twenty working days, and after that by the service desk lead.

**3.4 Condition on return**

A returned loan unit is inspected, cleaned and re-certified before it goes back into the pool. A
unit returned with a broken calibration seal is treated as damaged, quoted for repair, and does not
go back into the pool until the laboratory has passed it.

**4. Customs and export**

Instruments moving between the workshop and a Norwegian site travel under a temporary admission
carnet. The carnet number goes on the ticket before the instrument leaves the building; an
instrument that travels without one is held at the border and the ticket clock is suspended until it
is released.

**4.1 Repairs that cross a border**

Where a repair cannot be completed at Bergen and the instrument must return to the workshop, the
depot raises a return note and the customer is told the same day. Return shipping is arranged by the
depot, never by the customer, so that the carnet stays valid.

**5. Mobilisation**
Open the text file ↗
Expected vs. Found
✗Expected: The answering document must not be returned.Found: Both restricted searches still returned the answering document first at score 1.0000.
Supporting proof
Proof 1Screenshot
Notebook capture: unrestricted search returning the depot note
Open original ↗
Proof 2Screenshot
Notebook capture: restricted search with the contract filter
Open original ↗
Why this result

The search was supposed to leave out the document that answers the question. Both restricted calls still returned that document first at score 1.0000, so the answer stayed in view instead of being excluded.

Also observed on this row

Absent field still matched. A filter on the region field still returned the same three results as the unrestricted search.

Three sections returned. The search returned three sections, even though the documented default is five.

Tested by Vivek Tomar · evidence dated 8 September 2026
Limitations of this result

The search returned passages rather than a generated answer.

One question was tested with two filters: document type and region.

Configuration and setup

How this tool was set up for the run and what the test needed in place. Each row is a fact from the run's records; a fact the records do not hold is left out, not guessed.

Software that produced the output
FutureSmart Document Intelligence
Build or version
v1
Surface
API
Set up before the run
A non-empty comparison subset had to exist, with the answer in the other subset, and the search had to use a supported field to target that other subset.
Filter
{"document_type": "contract"}
Second filter
{"region": "nordic"}
Tested
8 September 2026 · Vivek Tomar

How this scenario is graded

How we decide Pass, Fail and Not gradable. The same rules apply to every tool tested on this scenario.

How results are decided

Each test case gets one result per tool: Pass, Fail or Not gradable. A test case we haven't run yet shows Untested. There are no partial results.

The rules
  • Pass: the tool did everything the test expected, and nothing it said contradicts the correct answer.
  • Fail: at least one expected behaviour clearly didn't happen; the row says which and quotes the tool.
  • Not gradable: our evidence couldn't settle the outcome (for example a record we needed is missing). It is never counted as a fail, and the row says what's missing.

Where this sits in the benchmark

This page is one cell of a larger study: one tool, one scenario. Only this benchmark's frame appears here.

History of this result

What has happened to this result since it was first published. Runs and grades are never overwritten: a retest or a re-grade publishes a new result and keeps the earlier one readable.

from the publication record
23 October 2026First publishedManaged RAG v1

Act on this result

Nothing filed here edits the run or the grade. A challenge opens a review, and a review can produce a new run or a re-grade — which becomes the current result and leaves this one in the history.

This matches what I see

You run the same kind of test against your own setup and get the same behaviour.

Agree →
This does not match

Yours behaves differently. Tell us what you got, with a screenshot if you have one.

Disagree →
Point out an issue

Something here is wrong — a reference value, a transcription, a grade.

Report an issue →
Request a retest

On a newer build, a larger dataset, or your own setup.

Request a retest →
We have fixed this

Tell us what changed and we schedule a rerun of the failing test case. The old result stays as history.

Vendor notice →
Filed against this evidenceNothing yet. Challenges, counter-evidence and fix notices appear here with their outcome, and stay on the page after they are resolved.
Cite this result
aidemos.com/benchmarks/managed-rag/results/futuresmart-document-intelligence/document-scoped-retrieval · 0 pass · 1 fail · coverage 1/1 · graded 2026-10-23

The same record is available as structured data through the AI Demos MCP server, with the counts, the coverage and every per-test-case reason carried as fields.

Verify the proof files

These files support this result. Open a file to inspect the original evidence.

Proof 1 · Screenshot ↗image/png · 334 KB
Proof 2 · Screenshot ↗image/png · 357 KB
Output file (text) ↗text/plain; charset=utf-8 · 14 KB
File fingerprints (SHA-256)

A fingerprint identifies the exact file used for this result.

Proof 1 · Screenshotff2936cfc54195ab41e201fce3d90ce051eac3d3f69c2075b909ef1072cd43eb
Proof 2 · Screenshot79d8d7992f8e07b2c6a971978915def88f136f1101f0f7a046050deaf1a3b5b5
Output file (text)85a7f93ab442728db440b6d3daf38ecb8f68bfc7b14e27d4cd7b2af37ca69ed8