Scenario in benchmark · Converting a complex PDF into clean Markdown with a hosted API · Version 1
A cleanly scanned document
A document is scanned and image-only, so its visible text must be recovered without relying on a text layer.
What we check
- Whether the visible text from the scanned page is recovered accurately.
- Whether the page is not silently skipped or treated as empty because no text layer exists.
- Whether the output stays faithful to the visible content without inventing text.
- Whether scanned pages are handled as real input, not hidden by omission.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
FirecrawlFailed1 of 1 test case failedDid not recover visible text accurately from the cleanly scanned document.View Result →
The output dropped the letterhead and footer, split words apart, interleaved the opening prose, kept the table values correct, and used full-width punctuation for the sampling time.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Nutrient.ioFailed1 of 1 test case failedDid not recover the text from a clean scan using its default conversion.View Result →
Although the API returned a successful response, the Markdown contained only two newline characters.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Extend AIPassed1 of 1 test case passedAccurately recovered text from a cleanly scanned, image-only document.View Result →
The one-page laboratory certificate was reproduced in Markdown with the letterhead, reference, date, six result rows, closing text, sign-off, and footer matching the page render. The results block also came back as a two-column HTML table with each label beside its value.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
GPT-5.6 terraPassed1 of 1 test case passedAccurately turned the clean, image-only certificate into Markdown without inventing text.View Result →
The letter's wording, figures and dates were preserved, and the handwritten signature was marked as an image rather than guessed.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Landing AIPassed1 of 1 test case passedPassed the clean-scan OCR scenario.View Result →
The image-only page had no text layer, and the returned Markdown preserves the letterhead, reference, report date, results rows with values and methods, and the footer in order; the handwritten signature stays a placeholder instead of a guessed name.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
LlamaParsePassed1 of 1 test case passedRecovered every printed word of the image-only certificate correctly, including the station code VQ1, all five mg/l readings, the four method codes and the dates.View Result →
The six-row results block came back as a table, and the only addition was a line standing in for the handwritten signature.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Mistral AIPassed1 of 1 test case passedAccurately recovered text from a clean scanned document.View Result →
It placed the results table’s first data row, the sampling point, in the Markdown header without losing text.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
ReductoPassed1 of 1 test case passedAccurately recovered the visible text from a cleanly scanned document.View Result →
The scan’s text matched the output field by field, with the handwritten signature left as a <signature> placeholder and only two middle-dot separators dropped from the letterhead and footer lines.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
UnstructuredPassed1 of 1 test case passedAccurately recovered text from a clean, image-only scan.View Result →
It preserved the certificate’s measurements, units and dates without adding text.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Upstage AIPassed1 of 1 test case passedRecovered the text from the scanned page.View Result →
The page had no text layer, and the Markdown returned every printed block correctly; the closing words were styled as a heading and the results came back as a pipe table.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (3): Adobe PDF Extract API, PDF Vector, Tensorlake.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 1 public · 0 withheld
- Capabilities exercised here
- Scanned Document OCR
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →