Scenario in benchmark · Managed RAG · Version 1

The retrieved sources disagree with each other

When retrieved sources give different values for the same fact.

Capability: Grounded Generation1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the answer recognises that the retrieved sources conflict on the same fact.
  2. Whether the answer avoids presenting one source’s value as the fact.
  3. Whether the answer conveys that the sources disagree instead of choosing a side without support.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 1
RAGFlowFailed1 of 1 test case failedChose one warranty period instead of flagging conflicting sources.View Result →

Its answer gave 24 months, although it had retrieved documents stating 24 and 36 months.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 3
Denser AIPassed1 of 1 test case passedConveyed that the retrieved sources disagreed.View Result →

For the warranty question, it gave both periods and recommended confirming with Fenwake Instruments which one applies.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

DifyPassed1 of 1 test case passedFlagged conflicting warranty terms rather than choosing one as fact.View Result →

It retrieved documents stating 24 months and 36 months from delivery and included both periods in its answer.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

FutureSmart Document IntelligencePassed1 of 1 test case passedShowed that the retrieved sources disagreed about the warranty period.View Result →

It reported 24 months from delivery in the customer-facing sales pack and 36 months from delivery in the service-desk service pack, with both periods beginning at delivery.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (4): Llama Cloud, Onyx, Vectara, Vectorize.io.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Grounded Generation

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

The retrieved sources disagree with each other in Managed RAG | AI Demos