Scenario in benchmark · Managed RAG · Version 1

Answer is available in the knowledge base

The relevant source already contains the answer, and the system surfaces it correctly.

Capability: Retrieval1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the system surfaces the answer when the relevant source already contains it.
  2. Whether the output is grounded in the source content rather than invented.
  3. Whether the system handles a present answer as a successful source-based result.

Specific test-case inputs and grading live on the related test-case pages.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Passed · 6
Denser AIPassed1 of 1 test case passedRetrieved the passage answering the question about Priority 1 faults reported outside business hours.View Result →

Its answer gave the policy’s acknowledgement target: within 90 minutes of the ticket being raised.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

DifyPassed1 of 1 test case passedPassed this retrieval task.View Result →

The answer-bearing passage was first in the results, and all four returned segments came from the same document.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

FutureSmart Document IntelligencePassed1 of 1 test case passedSurfaced the supporting passage for the knowledge-base answer.View Result →

The returned text matched the passage saying a Priority 1 fault reported outside business hours is acknowledged by a named duty engineer within 90 minutes, and the UI showed A01 first in Sources. Only four of eight source passages were visible in one capture, while the full notebook response was available in the attached text artifact.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

RAGFlowPassed1 of 1 test case passedRetrieved the passage answering the knowledge-base question.View Result →

The first result contained the 90-minute out-of-hours acknowledgement target; the business-hours passage ranked fifth.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

VectaraPassed1 of 1 test case passedReturned the answer from the knowledge base for this naturally phrased question.View Result →

The out-of-hours passage ranked first in three independent captures, including a live notebook run, while the business-hours near-miss stayed lower each time. The corpus view shows the relevant document was present.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Vectorize.ioPassed1 of 1 test case passedRetrieved the knowledge-base passage answering Fenwake’s out-of-hours Priority 1 question.View Result →

The passage ranked highest by relevance and specified acknowledgement within 90 minutes.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (2): Llama Cloud, Onyx.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Retrieval

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

Answer is available in the knowledge base in Managed RAG | AI Demos