Scenario in benchmark · Managed RAG · Version 1
Answer is available in the knowledge base
The relevant source already contains the answer, and the system surfaces it correctly.
What we check
- Whether the system surfaces the answer when the relevant source already contains it.
- Whether the output is grounded in the source content rather than invented.
- Whether the system handles a present answer as a successful source-based result.
Specific test-case inputs and grading live on the related test-case pages.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
Denser AIPassed1 of 1 test case passedRetrieved the passage answering the question about Priority 1 faults reported outside business hours.View Result →
Its answer gave the policy’s acknowledgement target: within 90 minutes of the ticket being raised.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
DifyPassed1 of 1 test case passedPassed this retrieval task.View Result →
The answer-bearing passage was first in the results, and all four returned segments came from the same document.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
FutureSmart Document IntelligencePassed1 of 1 test case passedSurfaced the supporting passage for the knowledge-base answer.View Result →
The returned text matched the passage saying a Priority 1 fault reported outside business hours is acknowledged by a named duty engineer within 90 minutes, and the UI showed A01 first in Sources. Only four of eight source passages were visible in one capture, while the full notebook response was available in the attached text artifact.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
RAGFlowPassed1 of 1 test case passedRetrieved the passage answering the knowledge-base question.View Result →
The first result contained the 90-minute out-of-hours acknowledgement target; the business-hours passage ranked fifth.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
VectaraPassed1 of 1 test case passedReturned the answer from the knowledge base for this naturally phrased question.View Result →
The out-of-hours passage ranked first in three independent captures, including a live notebook run, while the business-hours near-miss stayed lower each time. The corpus view shows the relevant document was present.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Vectorize.ioPassed1 of 1 test case passedRetrieved the knowledge-base passage answering Fenwake’s out-of-hours Priority 1 question.View Result →
The passage ranked highest by relevance and specified acknowledgement within 90 minutes.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (2): Llama Cloud, Onyx.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Retrieval
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →