Scenario in benchmark · Managed RAG · Version 1
Answer is not available in the knowledge base
The requested information is absent from the source set.
What we check
- Whether it indicates that the information cannot be found in the source set.
- Whether it avoids inventing an answer when the source set is missing the fact.
- Whether it responds with an honest refusal or equivalent no-answer behaviour.
Specific test-case inputs and grading live on the related test-case pages.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
FutureSmart Document IntelligenceFailed1 of 1 test case failedFails this scenario on the /search surface.View Result →
The unanswerable paid-parental-leave query returns three sections with a top score of 1.0000, and the returned handbook sections do not answer it. The control query on the same corpus also returns three sections with a top score of 1.0000, and its top section contains the answer. A separate /chat answer says the handbook does not specify any paid parental leave entitlement.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Denser AIPassed1 of 1 test case passedSignalled that the knowledge base had no answer to the paid parental leave question.View Result →
It did not invent an allowance from the related handbook material.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
RAGFlowPassed1 of 1 test case passedSignalled the missing answer through lower relevance scores, not an explicit no-match message.View Result →
Its ten returned passages did not address paid parental leave, and the highest score was 0.2865 versus 0.4826 for an answerable question.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
VectaraPassed1 of 1 test case passedSignaled the unavailable paid-parental-leave answer through clearly lower relevance scores.View Result →
In a second default-reranker call with no context window specified, the top score was 0.0820, versus 0.9986 for an answerable question on the same documents.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (4): Dify, Llama Cloud, Onyx, Vectorize.io.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Retrieval
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →