Scenario in benchmark · Managed RAG · Version 1
Existing content is changed
A previously indexed item is changed, and the system should answer with the updated value rather than the stale one.
What we check
- Whether the updated value becomes the answer after the source has been refreshed.
- Whether the stale value is no longer treated as current.
- Whether the system shows that it has caught up with the changed source rather than continuing to answer from old indexed content.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
Denser AIPassed1 of 1 test case passedAnswered with the updated Bergen shipping lead time of 6 working days rather than the previous 11 working days.View Result →
The identical question retrieved the revised shipping passage without the old value appearing in the returned passages.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
FutureSmart Document IntelligencePassed1 of 1 test case passedPassed this changed-content scenario.View Result →
Before the refresh it answered 11 working days; after the old B02 was deleted and the replacement ingested, it answered 6 working days, with the new file first and the old value absent.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (6): Dify, Llama Cloud, Onyx, RAGFlow, Vectara, Vectorize.io.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Knowledge Sync
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →