Scenario in benchmark · Converting a complex PDF into clean Markdown with a hosted API · Version 1
A page laid out in multiple columns
A PDF page laid out in multiple columns.
What we check
- Whether text is emitted in the natural reading order down each column.
- Whether content from different columns is kept separate rather than interleaved.
- Whether the page's reading sequence is preserved even when all text is present.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
FirecrawlFailed1 of 1 test case failedDid not consistently preserve column reading order.View Result →
In a three-page bulletin, it merged a left-column heading into a right-column heading on page 1; the remaining text followed the intended column order.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Nutrient.ioFailed1 of 1 test case failedDid not consistently preserve natural reading order for a two-column PDF in its default Build pipeline.View Result →
It placed a masthead line in the middle of a body sentence, although the body columns themselves were read in order on all three pages.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Adobe PDF Extract APIPassed1 of 1 test case passedPreserved the natural reading order of the two-column pages.View Result →
In the Markdown assembled from its response, sections followed the source columns and sentences continued across the column breaks on pages 1 and 2.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Extend AIPassed1 of 1 test case passedPreserved the reading order of a two-column bulletin.View Result →
It returned each page’s left column before its right column, without interleaving or inventing words.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Landing AIPassed1 of 1 test case passedReturned the two-column bulletin's text down each column in turn, without mixing lines from the two columns.View Result →
On pages 1 and 2, where a sentence runs from the left column into the right, its two halves came back joined with nothing in between, and each page's sections followed in left-column order, then right-column order.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
LlamaParsePassed1 of 1 test case passedRead the two-column bulletin in natural order on all three pages.View Result →
It finished each column before continuing to the next, without interleaving the text.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Mistral AIPassed1 of 1 test case passedRead the two-column page down each column in turn rather than across the page.View Result →
On the three pages, the left and right columns stayed in order without interleaving.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (6): GPT-5.6 terra, PDF Vector, Reducto, Tensorlake, Unstructured, Upstage AI.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 1 public · 0 withheld
- Capabilities exercised here
- Reading Order & Layout
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →