Scenario in benchmark · Managed RAG · Version 1
The retrieved material supports an answer
The retrieved material contains support for the answer.
What we check
- Whether the retrieved material contains enough support for the claim being made.
- Whether the system can ground its response in the retrieved material rather than outside knowledge.
- Whether the case is treated as a supported baseline, not as an unsupported or missing-answer case.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
Denser AIPassed1 of 1 test case passedAnswered from the retrieved policy, giving its 90-minute out-of-hours acknowledgement target.View Result →
The confirmation requirements and cover-level explanation were also supported by the passages.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
DifyPassed1 of 1 test case passedGenerated an answer supported by the retrieved material.View Result →
The reply matched the top retrieved passage, which said the fault is acknowledged within 90 minutes by a named duty engineer, and the lower retrieved passages did not contradict it.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
FutureSmart Document IntelligencePassed1 of 1 test case passedProduced an answer that is supported by the passages in view.View Result →
The checked passage supports the out-of-hours acknowledgement timing, named duty engineer, telephone ownership confirmation, and written confirmation on request, with no unsupported additions in the graded check.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
RAGFlowPassed1 of 1 test case passedGave a fully supported answer to the out-of-hours acknowledgement question.View Result →
It stated the policy’s 90-minute target, and the cited passage supported all three claims.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
VectaraPassed1 of 1 test case passedGave the supported 90-minute acknowledgement time for an out-of-hours fault.View Result →
It did not mix in the competing 30-minute target from another retrieved passage.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (3): Llama Cloud, Onyx, Vectorize.io.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Grounded Generation
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →