Scenario in benchmark · Converting a complex PDF into clean Markdown with an open-source library · Version 1
A document with figures and captions
A document contains a figure with its caption attached.
What we check
- Whether the figure is preserved rather than omitted or replaced by surrounding prose.
- Whether the figure remains positioned in the output at the point where it belongs in the document flow.
- Whether the caption stays attached to the figure and remains associated with it.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
DoclingFailed1 of 1 test case failedDocling’s default Markdown export omitted the figure images.View Result →
It kept the transect drawing’s caption and the bar chart’s title in their reading positions, with image placeholders rather than embedded pictures or image references.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
LiteParseFailed1 of 1 test case failedDid not preserve the captioned figure in Markdown.View Result →
It kept the caption unchanged and the reference in the right position, but embedded no image and wrote no image file.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
PyMuPDF4LLMFailed1 of 1 test case failedPyMuPDF4LLM’s default Markdown conversion omitted the figure image while keeping its caption in the right place.View Result →
A block of the figure’s text labels appeared where the image belonged, without an image or image link.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (7): doc2mark, GPT-5.6 terra, Marker, MarkItDown, MinerU, olmOCR, PaddleOCR-VL.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 1 public · 0 withheld
- Capabilities exercised here
- Figures & Charts
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →