Scenario in benchmark · Managed RAG · Version 1
New content is added to the source
After new content is added and the refresh path completes, the updated index should make the new fact findable.
What we check
- Whether a query after refresh can retrieve the added content from the updated index.
- Whether the tool completes its normal refresh or update path before the result is judged.
- Whether the refreshed index still behaves as if the added content is absent.
- Whether the system exposes the new fact through retrieval rather than only in the source.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
Denser AIPassed1 of 1 test case passedReturned a fact from newly added content after the source was refreshed.View Result →
Before the addition, it could not answer the same question; afterwards, it gave the depot opening date and cited the new bulletin.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
DifyPassed1 of 1 test case passedMade newly added source content findable after the update path completed.View Result →
The new document was created through the ingestion endpoint, indexing reached completed before retrieval, and the retrieval call returned that same document first; two additional segments also pointed to it.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
FutureSmart Document IntelligencePassed1 of 1 test case passedFound the newly added content after it was indexed.View Result →
Before the add, it said the retrieved information did not state when the Tromso depot opens; after the bulletin was indexed, it answered 14 October 2026 and ranked that bulletin first.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Vectorize.ioPassed1 of 1 test case passedMade newly added content findable after the update path finished.View Result →
Before the add, the question scored almost zero; after the refresh, retrieval returned the added bulletin and the answer gave October 14, 2026.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (4): Llama Cloud, Onyx, RAGFlow, Vectara.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Knowledge Sync
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →