Scenario in benchmark · Managed RAG · Version 1

A cleanly scanned document

A document is scanned and image-only, so its visible text must be recovered without relying on a text layer.

Capability: Data Ingestion1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the visible text from the scanned page is recovered accurately.
  2. Whether the page is not silently skipped or treated as empty because no text layer exists.
  3. Whether the output stays faithful to the visible content without inventing text.
  4. Whether scanned pages are handled as real input, not hidden by omission.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 1
DifyFailed1 of 1 test case failedDid not turn the clean scan into accurate text.View Result →

The scan had no extractable text, yet the document was marked completed and available with a word count, and the lookup returned other records instead of the scan's fact; the Pipeline view was present but unopened.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 2
FutureSmart Document IntelligencePassed1 of 1 test case passedReturned the OCR text from the cleanly scanned document.View Result →

It returned the as-found bias, -0.037 psu, for certificate C-77413; separately, the scanned certificate was the top search hit.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

RAGFlowPassed1 of 1 test case passedRecovered the requested fact from an image-only calibration certificate.View Result →

Its first-ranked result contained an as-found bias of -0.037 psu, matching the reference value of -0.037 psu.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (5): Denser AI, Llama Cloud, Onyx, Vectara, Vectorize.io.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Data Ingestion

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

A cleanly scanned document in Managed RAG | AI Demos