Scenario in benchmark · Converting a complex PDF into clean Markdown with a hosted API · Version 1

A document containing a code block

A document contains a code block.

Capability: Code Extraction1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the code remains a distinct preformatted or fenced block.
  2. Whether line breaks and indentation are preserved.
  3. Whether the code is not reflowed into ordinary prose.
  4. Whether the code block is still recognisable as code rather than flattened text.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 8
Adobe PDF Extract APIFailed1 of 1 test case failedDid not preserve the code block as a preformatted or fenced block.View Result →

The nine-line Python block was collapsed into five ordinary paragraphs, the shell block into two lines, and no output line carried leading whitespace. The whitespace-insensitive character content still matched the source, but the block structure was lost.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Extend AIFailed1 of 1 test case failedFlattened the PDF’s two code snippets into ordinary text rather than preserving their line structure and indentation.View Result →

Code characters also changed, and neither snippet would run as returned.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

FirecrawlFailed1 of 1 test case failedDid not preserve the full structure of the document’s code blocks.View Result →

It fenced both snippets and retained their code characters, but removed their indentation.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Mistral AIFailed1 of 1 test case failedReturned code without fences and flattened its nested indentation when converting the PDF to Markdown.View Result →

The Python code was no longer runnable, and the shell command lost its continuation indentation.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Nutrient.ioFailed1 of 1 test case failedDid not preserve the code block in its default PDF-to-Markdown conversion.View Result →

It returned the Python snippet as a table without a code fence or its original indentation.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

PDF VectorFailed1 of 1 test case failedReturned the code without code fences or indentation in Markdown.View Result →

The listing ran directly into the following prose.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

ReductoFailed1 of 1 test case failedDid not preserve the code block as fenced or preformatted Markdown.View Result →

The returned file had no fences or indented code lines, while the source block’s line breaks remained and the output also altered code tokens, including `window = reload_window(reset_at, margin_minutes=45)` becoming `window reload_window (reset_at, margin_minutes=45)`.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Upstage AIFailed1 of 1 test case failedDid not preserve code-block formatting when converting the PDF to Markdown.View Result →

Its output kept the code's characters and line breaks, but lost indentation and treated the code as ordinary paragraphs.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 1
LlamaParsePassed1 of 1 test case passedReturned both code listings in the document as fenced blocks tagged as Python and shell, and kept the Python body at its three indent levels of 4, 8 and 12 spaces.View Result →

Two small whitespace differences remain inside the fences: one blank line was dropped, and the shell continuation line sits at four spaces rather than six.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (4): GPT-5.6 terra, Landing AI, Tensorlake, Unstructured.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
1 public · 0 withheld
Capabilities exercised here
Code Extraction

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

A document containing a code block in Converting a complex PDF into clean Markdown with a hosted API | AI Demos