Scenario in benchmark · Managed RAG · Version 1
Content is removed from the source
Content that was already returned successfully is removed from the source.
What we check
- Whether the content was returned successfully before it was removed from the source.
- Whether the removed content is still returned after the source no longer contains it.
- Whether the system stops surfacing the removed content once removal has taken effect.
Specific test-case inputs and grading live on the related test-case pages.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
Denser AIPassed1 of 1 test case passedDeclined to give a supplier after the source document was deleted.View Result →
Previously it had named Ardwyn Fabrication; afterwards, the supplier list appeared in neither retrieved passages nor citations.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
FutureSmart Document IntelligencePassed1 of 1 test case passedAfter deletion and refresh, FutureSmart Document Intelligence no longer returned the removed document in the query results or citations, and it said the retrieved documents do not identify an approved supplier for the H-clamp bracket.View Result →
The evidence shows four surviving sections in the after capture, with the deleted document absent from them.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
RAGFlowPassed1 of 1 test case passedStopped using the supplier list after it was deleted.View Result →
It had previously answered the supplier question with a citation to that list; afterwards, it said the information was not found and gave no citations.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
VectaraPassed1 of 1 test case passedStopped using the deleted source and declined to answer the question.View Result →
Across four repeat queries, the deleted document was absent from the top ten results and the supplier was not cited.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (4): Dify, Llama Cloud, Onyx, Vectorize.io.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Knowledge Sync
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →