Scenario in benchmark · Converting a complex PDF into clean Markdown with an open-source library · Version 1

A document containing a code block

A document contains a code block.

Capability: Code Extraction1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the code remains a distinct preformatted or fenced block.
  2. Whether line breaks and indentation are preserved.
  3. Whether the code is not reflowed into ordinary prose.
  4. Whether the code block is still recognisable as code rather than flattened text.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 1
DoclingFailed1 of 1 test case failedFailed to keep code block line structure and indentation intact.View Result →

It fenced the Python, but the code and shell blocks collapsed to single lines, and the block-header excerpt became ordinary prose.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 1
PyMuPDF4LLMPassed1 of 1 test case passedKeeps the function as fenced Markdown with its line breaks and indentation intact.View Result →

The function appears in its own fence with eight code lines and no tabs; the import line is fenced separately, and the blank lines before the loop and return are omitted.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (8): doc2mark, GPT-5.6 terra, LiteParse, Marker, MarkItDown, MinerU, olmOCR, PaddleOCR-VL.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
1 public · 0 withheld
Capabilities exercised here
Code Extraction

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

A document containing a code block in Converting a complex PDF into clean Markdown with an open-source library | AI Demos