Scenario in benchmark · Managed RAG · Version 1
The request uses different wording than the source
A request uses wording different from the source, but the system must still resolve the same underlying fact or passage.
What we check
- Whether the system recognises the same underlying fact or passage despite different wording.
- Whether it relies on meaning rather than an exact phrase match.
- Whether it still resolves the correct source when the request is paraphrased.
- Whether it treats different wording as equivalent when the underlying meaning is the same.
Specific test-case inputs and grading live on the related test-case pages.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
Denser AIPassed1 of 1 test case passedHandled the wording mismatch and returned the supporting handbook passage.View Result →
The handbook ranked first, the answer repeated its 26-day annual-leave figure with a citation, and the alternative phrase appeared only in the question.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
DifyPassed1 of 1 test case passedRetrieved the supporting passage when asked about paid time off, while the employee handbook calls it annual leave.View Result →
The passage gives full-time employees 26 days per complete leave year.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
FutureSmart Document IntelligencePassed1 of 1 test case passedHandled the wording change and still returned the annual-leave answer.View Result →
Search ranked the handbook passage first, and chat answered 26 days per complete leave year with part-time pro rata.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
RAGFlowPassed1 of 1 test case passedRetrieved the supporting passage despite the question using different wording from the source.View Result →
It ranked the annual-leave passage first in both requests.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
VectaraPassed1 of 1 test case passedReturned the matching passage even though the question used different wording from the source.View Result →
In both retrieval runs, the annual leave passage was rank 1, and the returned passages did not contain the query's alternative terms.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (3): Llama Cloud, Onyx, Vectorize.io.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Retrieval
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →