Scenario in benchmark · Converting a complex PDF into clean Markdown with an open-source library · Version 1

A document with styled headings and subheadings

A PDF document containing styled headings and subheadings.

Capability: Heading & Section Structure1 pinned test case per toolNeutral global definition →

What we check

  1. Whether headings are emitted as Markdown heading levels rather than plain or bold paragraphs.
  2. Whether nested headings keep their parent-child hierarchy intact.
  3. Whether the document’s organisation remains visibly structured instead of flattening to one level.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 1
DoclingFailed1 of 1 test case failedDid not keep the heading hierarchy intact for this PDF.View Result →

The source showed three heading levels and a title block, but the Markdown flattened all 26 headings to level 2 and merged one section with its subsection.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 2
LiteParsePassed1 of 1 test case passedKept the styled headings nested correctly in Markdown.View Result →

The source has three heading tiers, and the output matches them in count, strings and order; the title block remains plain text.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

PyMuPDF4LLMPassed1 of 1 test case passedKept the heading hierarchy in this PDF-to-Markdown run.View Result →

It produced 15 #, 9 ## and 2 ### lines in source order; the three body-sized title lines were emitted as ####, but that false positive did not change the real heading structure.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (7): doc2mark, GPT-5.6 terra, Marker, MarkItDown, MinerU, olmOCR, PaddleOCR-VL.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
1 public · 0 withheld
Capabilities exercised here
Heading & Section Structure

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

A document with styled headings and subheadings in Converting a complex PDF into clean Markdown with an open-source library | AI Demos