Scenario in benchmark · Converting a complex PDF into clean Markdown with a hosted API · Version 1

A document mixing digital and scanned pages

A document contains both digital text pages and scanned image-only pages.

Capability: Scanned Document OCR1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the scanned page inside a mixed document is processed rather than silently skipped.
  2. Whether page-by-page handling keeps content from both digital text pages and scanned image-only pages.
  3. Whether the system treats the document as mixed, not as a fully digital file with no special handling needed.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Passed · 8
Adobe PDF Extract APIPassed1 of 1 test case passedRecovered the scanned page of a three-page PDF in full: the page has no text layer, yet its readings, site times and sign-off rows came back in the same table form as the digital pages, every value matching the scan.View Result →

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Extend AIPassed1 of 1 test case passedConverted the scanned worksheet alongside the digital pages without skipping it.View Result →

The worksheet's readable values stayed in place, with nothing added.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

FirecrawlPassed1 of 1 test case passedProcessed the scanned page in a mixed digital and scanned PDF.View Result →

It returned the worksheet in document order rather than skipping it.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Landing AIPassed1 of 1 test case passedProcessed the scanned page as well as the digital pages.View Result →

Its Markdown included every worksheet value in the original document position.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

LlamaParsePassed1 of 1 test case passedProcessed the scanned page in a PDF that also contained digital pages.View Result →

Its Markdown included the scanned worksheet’s text, readings, identifiers and times.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Mistral AIPassed1 of 1 test case passedProcessed the scanned page in a mixed digital and scanned document rather than skipping it.View Result →

The page appeared in document order, with tables in the same style as the digital pages.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

ReductoPassed1 of 1 test case passedThe mixed PDF’s scanned page was processed rather than skipped.View Result →

Reducto’s Markdown includes the image-only page 2 with its headers, table, certificate section, and recipient details, matching the digital pages’ presentation in the checked evidence.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Upstage AIPassed1 of 1 test case passedProcessed the scanned page inside the mixed PDF instead of skipping it.View Result →

The scanned worksheet appears in place between the two digital pages, and the service reports all three pages processed with OCR on; the scanned page also returns the most elements, with only small spacing and punctuation changes.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (5): GPT-5.6 terra, Nutrient.io, PDF Vector, Tensorlake, Unstructured.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
1 public · 0 withheld
Capabilities exercised here
Scanned Document OCR

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

A document mixing digital and scanned pages in Converting a complex PDF into clean Markdown with a hosted API | AI Demos