New content is added to the source
After new content is added and the refresh path completes, the updated index should make the new fact findable.
What this scenario means
This tests whether a managed knowledge system catches up after source content changes. A good system finishes its normal refresh or update path and then retrieves the added fact from the refreshed index, instead of continuing to behave as if the addition were absent. The hard part is that the failure can look like silence: the content is present in the source, but not yet reachable through query.
What we evaluate
- Whether a query after refresh can retrieve the added content from the updated index.
- Whether the tool completes its normal refresh or update path before the result is judged.
- Whether the refreshed index still behaves as if the added content is absent.
- Whether the system exposes the new fact through retrieval rather than only in the source.
Capabilities this scenario exercises
A scenario may exercise one or more capabilities.
Knowledge Sync
The indexed state converges on the source after content is added, changed or removed. Each test follows the subject's own documented refresh path to completion; elapsed time is recorded, convergence is what is graded, and a path that never completes is recorded as non-convergence rather than as a large duration.
Benchmarks that use this scenario
A scenario has global identity and may be reused across benchmarks.
Managed RAG
Which managed RAG service turns a folder of private documents into a query API that finds the right passage, answers only from what it retrieved, cites the source that actually supports the claim — and keeps up when the source changes?