Scenario in benchmark · Managed RAG · Version 1

A claim in the answer traced back to its source

A citation in an answer lands on text that supports the claim it is attached to.

Capability: Citations1 pinned test case per toolNeutral global definition →

What we check

  1. Whether each citation resolves to text that supports the specific claim it accompanies.
  2. Whether a citation that points to the right source but not the supporting passage is treated as incorrect.
  3. Whether a citation that lands on related text without claim support is treated as incorrect.
  4. Whether the cited passage can be checked directly against the claim without relying on document-level similarity alone.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 2
FutureSmart Document IntelligenceFailed1 of 1 test case failedDoes not link the answer claim to a supporting passage.View Result →

The chat response had no citation marker or section reference, and the web Conversations view panel listed source files rather than the supporting passage; the top section matched the answer, but a conflicting 30-minute section also appeared.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Vectorize.ioFailed1 of 1 test case failedFailed to give a citation that opened to text supporting the answer's claim.View Result →

Both executions showed no citation pointer, so nothing could be opened against the claim.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 4
Denser AIPassed1 of 1 test case passedTraced the answer’s claim to supporting text in the cited passage.View Result →

The marker landed on the passage with the 90-minute acknowledgement statement and its cover-level note, while the decoy table in another passage was not cited.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

DifyPassed1 of 1 test case passedPassed this citation check.View Result →

The answer said acknowledgement would happen within 90 minutes by a named duty engineer, and the top citation pointed to that same passage. The other citations also came from the same policy.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

RAGFlowPassed1 of 1 test case passedRAGFlow’s out-of-hours Priority 1 answer cited text supporting all four claims, including acknowledgement within 90 minutes of the ticket being raised.View Result →

The retrieved table giving 30 minutes was not cited.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

VectaraPassed1 of 1 test case passedCited a passage that supported its answer's claim.View Result →

It said a named duty engineer would acknowledge the out-of-hours fault within 90 minutes, and the cited passage said the same.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (2): Llama Cloud, Onyx.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Citations

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

A claim in the answer traced back to its source in Managed RAG | AI Demos