Scenario in benchmark · Converting a complex PDF into clean Markdown with an open-source library · Version 1
A document with a data chart
A document includes a data chart that must be converted as part of the page.
What we check
- Whether the chart remains present as a visual asset in the converted output.
- Whether the chart stays associated with its original position and its title or caption.
- Whether any extra textual chart details are treated as optional enrichment, not as the minimum bar.
- Whether the visual chart is not replaced by prose alone.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
DoclingFailed1 of 1 test case failedDocling’s default Markdown conversion did not preserve the data chart as an image.View Result →
The chart title became a heading, followed by an image placeholder at the correct position.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
LiteParseFailed1 of 1 test case failedDid not keep the chart as an image in Markdown.View Result →
It preserved the chart's title and position, but the output reduced the chart to a placeholder link, with no image file written and OCR text standing in for the chart content.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
PyMuPDF4LLMFailed1 of 1 test case failedDid not preserve the chart as an image in its default conversion.View Result →
It retained the title but omitted the twelve plotted values. In a separate run with image embedding enabled, it emitted the chart as a PNG with its title and all twelve bars.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (7): doc2mark, GPT-5.6 terra, Marker, MarkItDown, MinerU, olmOCR, PaddleOCR-VL.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 1 public · 0 withheld
- Capabilities exercised here
- Figures & Charts
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →