Scenario in benchmark · Managed RAG · Version 1

An ordinary digital text document

A plain digital text document used as the baseline text-only document condition.

Capability: Data Ingestion1 pinned test case per toolNeutral global definition →

What we check

  1. Whether every paragraph is preserved complete and unchanged.
  2. Whether nothing is silently missing, garbled, or invented.
  3. Whether the result remains recognisably plain text without unintended changes to the content.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Passed · 5
Denser AIPassed1 of 1 test case passedReturned the requested fact from the ordinary text document.View Result →

The 22-month interval came back as the top passage, and the answer repeated the same value; the other returned passages did not state that interval.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

DifyPassed1 of 1 test case passedReturned the passage that states the asked-for fact from a plain digital text document.View Result →

Asked for the Tolvan A3's recommended recalibration interval in the document's own words, it ranked the whole Calibration passage first, which gives the interval as 22 months from the date of the last certificate.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

FutureSmart Document IntelligencePassed1 of 1 test case passedReturned the calibration passage first, and the answer gave 22 months from the last calibration certificate.View Result →

A 36-month warranty passage also appeared at a lower score and did not replace the calibration passage.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

RAGFlowPassed1 of 1 test case passedPasses this plain-text document scenario.View Result →

The ingested document contains the target sentence, and the default retrieval call returns that chunk first with a clear score gap.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

VectaraPassed1 of 1 test case passedReturned the requested fact in the document's own words.View Result →

In both configurations, the top result was the passage stating a 22-month recalibration interval, and the next results were different warranty passages; both captures showed HTTP 200.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (3): Llama Cloud, Onyx, Vectorize.io.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Data Ingestion

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

An ordinary digital text document in Managed RAG | AI Demos