Scenario in benchmark · Converting a complex PDF into clean Markdown with an open-source library · Version 1
An ordinary digital text document
A plain digital text document used as the baseline text-only document condition.
What we check
- Whether every paragraph is preserved complete and unchanged.
- Whether nothing is silently missing, garbled, or invented.
- Whether the result remains recognisably plain text without unintended changes to the content.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
DoclingFailed1 of 1 test case failedOmitted four body words when converting the digital document to Markdown.View Result →
The next paragraph was joined to an unfinished sentence; nothing was garbled, invented or duplicated.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
PyMuPDF4LLMFailed1 of 1 test case failedFails to keep an ordinary digital text document complete and unchanged.View Result →
It keeps the words, but it invents structure at page breaks: a body continuation becomes a level-3 heading with underline markup, and running headers are folded into body sentences.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (8): doc2mark, GPT-5.6 terra, LiteParse, Marker, MarkItDown, MinerU, olmOCR, PaddleOCR-VL.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 1 public · 0 withheld
- Capabilities exercised here
- Text Fidelity
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →