Scenario in benchmark · Managed RAG · Version 1

Retrieval is scoped to part of the documents

When retrieval is limited to a defined subset of documents, it should stay inside that scope.

Capability: Retrieval1 pinned test case per toolNeutral global definition →

What we check

  1. Whether returned results stay within the allowed subset when search is restricted.
  2. Whether the answer-bearing document outside the allowed subset is excluded from retrieval.
  3. Whether the system treats a narrow scope as a true no-result case rather than a successful hit.
  4. Whether the restriction is respected instead of being silently ignored.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 1
FutureSmart Document IntelligenceFailed1 of 1 test case failedFailed to keep retrieval scoped to the subset that did not hold the answer.View Result →

Both restricted searches still returned the answering depot note first at score 1.0000.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 1
RAGFlowPassed1 of 1 test case passedRespected the document restriction, returning three Nordic service passages without the depot note that held the answer.View Result →

The depot note ranked first in both the unrestricted search and the UK-only comparison search.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (6): Denser AI, Dify, Llama Cloud, Onyx, Vectara, Vectorize.io.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Retrieval

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

Retrieval is scoped to part of the documents in Managed RAG | AI Demos