Content is removed from the source
Content that was already returned successfully is removed from the source.
What this scenario means
This tests whether content that was previously returned successfully and then removed is still surfaced afterwards. The correct behaviour is to stop returning the removed content once it has been deleted from the source, so the system no longer shows it as available.
What we evaluate
- Whether the content was returned successfully before it was removed from the source.
- Whether the removed content is still returned after the source no longer contains it.
- Whether the system stops surfacing the removed content once removal has taken effect.
Capabilities this scenario exercises
A scenario may exercise one or more capabilities.
Freshness
Does what it returns match the web as it is now — new pages found, changed pages not served stale, deleted pages not still handed to you? Three independent failure directions; a tool can be good at one and bad at another. Every tool must be tested against the same page or event. Every run is probe-before, the world changes, probe-after; a run whose before-probe comes out the wrong way is VOID, not failed.
Knowledge Sync
The indexed state converges on the source after content is added, changed or removed. Each test follows the subject's own documented refresh path to completion; elapsed time is recorded, convergence is what is graded, and a path that never completes is recorded as non-convergence rather than as a large duration.
Benchmarks that use this scenario
A scenario has global identity and may be reused across benchmarks.
Managed RAG
Which managed RAG service turns a folder of private documents into a query API that finds the right passage, answers only from what it retrieved, cites the source that actually supports the claim — and keeps up when the source changes?
Web Search for AI
Also uses this scenario.