The retrieved sources disagree with each other
When retrieved sources give different values for the same fact.
What this scenario means
This scenario checks whether an agent notices conflict in the material it retrieved and resists turning one source into the answer by default. It is hard because both sources can look usable, but a good agent must surface the disagreement and avoid stating one value as settled fact.
What we evaluate
- Whether the answer recognises that the retrieved sources conflict on the same fact.
- Whether the answer avoids presenting one source’s value as the fact.
- Whether the answer conveys that the sources disagree instead of choosing a side without support.
Capabilities this scenario exercises
A scenario may exercise one or more capabilities.
Answer Generation
Where it writes the answer, does it say only what its own sources support -- declining when they don't, flagging when they disagree, and attributing what it does claim? Judged against what THAT TOOL retrieved, never against the web. Out of scope for search-only products, which is not a zero and never drags them down a ranking. A citation that correctly points at the source of a wrong answer is a CORRECT citation.
Grounded Generation
The answer is faithful to the material the system itself retrieved, including declining when that material doesn't support one.
Benchmarks that use this scenario
A scenario has global identity and may be reused across benchmarks.
Managed RAG
Which managed RAG service turns a folder of private documents into a query API that finds the right passage, answers only from what it retrieved, cites the source that actually supports the claim — and keeps up when the source changes?
Web Search for AI
Also uses this scenario.