Scenario in benchmark · Converting a complex PDF into clean Markdown with a hosted API · Version 1

An ordinary digital text document

A plain digital text document used as the baseline text-only document condition.

Capability: Text Fidelity1 pinned test case per toolNeutral global definition →

What we check

  1. Whether every paragraph is preserved complete and unchanged.
  2. Whether nothing is silently missing, garbled, or invented.
  3. Whether the result remains recognisably plain text without unintended changes to the content.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 8
Adobe PDF Extract APIFailed1 of 1 test case failedDid not keep the document text unchanged.View Result →

Its Extract output joined the running header to the document title and to a sentence continuing onto page 2; no words were missing or invented.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

FirecrawlFailed1 of 1 test case failedKept the body words but broke a paragraph by turning a sentence continuation into a heading.View Result →

The change occurred at the break between pages 1 and 2 of the 7-page document.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Landing AIFailed1 of 1 test case failedReturned all 3,122 body words of the seven-page document in source order, with nothing missing or invented.View Result →

But at four of the six page joins it merged the running page header into the body, so those passages read as sentences the document does not contain.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Mistral AIFailed1 of 1 test case failedDid not keep ordinary digital text complete and unchanged.View Result →

The returned Markdown preserves the body prose, but page furniture lands inside the text at four page breaks, and the raw page markdown shows the same split.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Nutrient.ioFailed1 of 1 test case failedFailed on ordinary digital text.View Result →

The returned Markdown kept the words, but four page breaks merged the running header into body text, while two breaks stayed clean.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

PDF VectorFailed1 of 1 test case failedReturned every body word of a seven-page briefing note in order and invented nothing, but it put the running header and the 'Page N of 7' line inline at every page boundary.View Result →

At four of the six boundaries they land inside a sentence broken across the page, so the returned text has sentences the source does not.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

UnstructuredFailed1 of 1 test case failedFails to keep an ordinary digital text document complete and unchanged.View Result →

The returned Markdown matches the body text after page furniture is removed, but all seven page footers become headings and four body sentences are split by running-header words.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Upstage AIFailed1 of 1 test case failedDid not keep ordinary digital text unchanged.View Result →

At one page break, it joined a running header to a continuation of the body text; the other five breaks stayed clean.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 2
Extend AIPassed1 of 1 test case passedPasses the ordinary digital text scenario.View Result →

The returned Markdown keeps the source text in order with no inserted or missing body words; one middle dot becomes a period, and running headers and page numbers stay separate from the body.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

LlamaParsePassed1 of 1 test case passedPreserved the briefing note’s body text word for word: 3,122 words in the source and 3,122 in Markdown.View Result →

All eleven money and measurement figures were present.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (3): GPT-5.6 terra, Reducto, Tensorlake.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
1 public · 0 withheld
Capabilities exercised here
Text Fidelity

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

An ordinary digital text document in Converting a complex PDF into clean Markdown with a hosted API | AI Demos