Scenario in benchmark · Managed RAG · Version 1
The retrieved material doesn't support an answer
Retrieved material is related to the topic but does not support an answer.
What we check
- Whether the response withholds a direct answer when the retrieved material is insufficient.
- Whether the response avoids presenting related material as if it supports the conclusion.
- Whether the response indicates that the retrieved material does not answer the question, rather than overclaiming confidence.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
Denser AIPassed1 of 1 test case passedDeclined the parental-leave question instead of inventing an answer.View Result →
The returned passages were handbook, warranty, and field-service material with no parental-leave details, and the answer gave no figure or policy while directing the customer to HR and statutory provisions.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
DifyPassed1 of 1 test case passedHandled the unsupported question by saying it did not know and giving no leave amount or policy.View Result →
The retrieved passages were unrelated to parental leave and instead covered handbook and field-service material.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
FutureSmart Document IntelligencePassed1 of 1 test case passedDeclined to answer the unsupported paid parental leave question and gave no paid-leave figure.View Result →
Its reply said the employee handbook did not specify any paid parental leave entitlement, and retrieval returned 27 sections with a top score of 1.0000.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
RAGFlowPassed1 of 1 test case passedDeclined to answer when the retrieved material did not support an answer.View Result →
Across four responses to a paid parental leave question, it said the information was unavailable and supplied no entitlement figure or policy.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (4): Llama Cloud, Onyx, Vectara, Vectorize.io.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Grounded Generation
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →