Scenario in benchmark · Managed RAG · Version 1

Answer is not available in the knowledge base

The requested information is absent from the source set.

Capability: Retrieval1 pinned test case per toolNeutral global definition →

What we check

  1. Whether it indicates that the information cannot be found in the source set.
  2. Whether it avoids inventing an answer when the source set is missing the fact.
  3. Whether it responds with an honest refusal or equivalent no-answer behaviour.

Specific test-case inputs and grading live on the related test-case pages.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 1
FutureSmart Document IntelligenceFailed1 of 1 test case failedFails this scenario on the /search surface.View Result →

The unanswerable paid-parental-leave query returns three sections with a top score of 1.0000, and the returned handbook sections do not answer it. The control query on the same corpus also returns three sections with a top score of 1.0000, and its top section contains the answer. A separate /chat answer says the handbook does not specify any paid parental leave entitlement.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 3
Denser AIPassed1 of 1 test case passedSignalled that the knowledge base had no answer to the paid parental leave question.View Result →

It did not invent an allowance from the related handbook material.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

RAGFlowPassed1 of 1 test case passedSignalled the missing answer through lower relevance scores, not an explicit no-match message.View Result →

Its ten returned passages did not address paid parental leave, and the highest score was 0.2865 versus 0.4826 for an answerable question.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

VectaraPassed1 of 1 test case passedSignaled the unavailable paid-parental-leave answer through clearly lower relevance scores.View Result →

In a second default-reranker call with no context window specified, the top score was 0.0820, versus 0.9986 for an answerable question on the same documents.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (4): Dify, Llama Cloud, Onyx, Vectorize.io.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Retrieval

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

Answer is not available in the knowledge base in Managed RAG | AI Demos