Scenario in benchmark · Managed RAG · Version 1

The request uses different wording than the source

A request uses wording different from the source, but the system must still resolve the same underlying fact or passage.

Capability: Retrieval1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the system recognises the same underlying fact or passage despite different wording.
  2. Whether it relies on meaning rather than an exact phrase match.
  3. Whether it still resolves the correct source when the request is paraphrased.
  4. Whether it treats different wording as equivalent when the underlying meaning is the same.

Specific test-case inputs and grading live on the related test-case pages.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Passed · 5
Denser AIPassed1 of 1 test case passedHandled the wording mismatch and returned the supporting handbook passage.View Result →

The handbook ranked first, the answer repeated its 26-day annual-leave figure with a citation, and the alternative phrase appeared only in the question.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

DifyPassed1 of 1 test case passedRetrieved the supporting passage when asked about paid time off, while the employee handbook calls it annual leave.View Result →

The passage gives full-time employees 26 days per complete leave year.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

FutureSmart Document IntelligencePassed1 of 1 test case passedHandled the wording change and still returned the annual-leave answer.View Result →

Search ranked the handbook passage first, and chat answered 26 days per complete leave year with part-time pro rata.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

RAGFlowPassed1 of 1 test case passedRetrieved the supporting passage despite the question using different wording from the source.View Result →

It ranked the annual-leave passage first in both requests.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

VectaraPassed1 of 1 test case passedReturned the matching passage even though the question used different wording from the source.View Result →

In both retrieval runs, the annual leave passage was rank 1, and the returned passages did not contain the query's alternative terms.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (3): Llama Cloud, Onyx, Vectorize.io.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Retrieval

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

The request uses different wording than the source in Managed RAG | AI Demos