Existing content is changed
A previously indexed item is changed, and the system should answer with the updated value rather than the stale one.
What this scenario means
This tests whether the indexed view catches up after the source changes. A stale index can keep serving an old answer even when the source has been edited, so the hard part is not finding something once but staying current after change. A good system follows its refresh path and reflects the present source rather than the outdated copy.
What we evaluate
- Whether the updated value becomes the answer after the source has been refreshed.
- Whether the stale value is no longer treated as current.
- Whether the system shows that it has caught up with the changed source rather than continuing to answer from old indexed content.
Capabilities this scenario exercises
A scenario may exercise one or more capabilities.
Knowledge Sync
The indexed state converges on the source after content is added, changed or removed. Each test follows the subject's own documented refresh path to completion; elapsed time is recorded, convergence is what is graded, and a path that never completes is recorded as non-convergence rather than as a large duration.
Benchmarks that use this scenario
A scenario has global identity and may be reused across benchmarks.
Managed RAG
Which managed RAG service turns a folder of private documents into a query API that finds the right passage, answers only from what it retrieved, cites the source that actually supports the claim — and keeps up when the source changes?