Scenario in benchmark · Converting a complex PDF into clean Markdown with a hosted API · Version 1

A document with styled headings and subheadings

A PDF document containing styled headings and subheadings.

Capability: Heading & Section Structure1 pinned test case per toolNeutral global definition →

What we check

  1. Whether headings are emitted as Markdown heading levels rather than plain or bold paragraphs.
  2. Whether nested headings keep their parent-child hierarchy intact.
  3. Whether the document’s organisation remains visibly structured instead of flattening to one level.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 7
Extend AIFailed1 of 1 test case failedDid not preserve the heading hierarchy in this PDF-to-Markdown conversion.View Result →

In the one tested case, all 26 headings became Markdown headings, but every one was level three, so the three source tiers were flattened into a single level.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

GPT-5.6 terraFailed1 of 1 test case failedTurned every heading into a real Markdown heading but flattened the hierarchy: the source has three heading tiers, and the output uses only two levels below the title.View Result →

The smallest tier came out at the same level as the middle one, and Chain of custody and Condition on arrival were lifted to the level of their parent, Reception.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Landing AIFailed1 of 1 test case failedDid not preserve heading hierarchy for this styled PDF-to-Markdown case.View Result →

The Markdown flattened most headings to level 1, kept only two at level 2, and turned one heading into plain body text; page 3 matched because that page had no subheadings.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

LlamaParseFailed1 of 1 test case failedTurned every heading into a Markdown heading but kept only two of the source's three heading levels.View Result →

On page 1, the subsections Chain of custody and Condition on arrival came out at the same level as their parent, Reception, and the third-tier headings shared a level with the second. A separate run with a custom prompt asking for three heading levels did not preserve the hierarchy either.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Nutrient.ioFailed1 of 1 test case failedNutrient.io's Build API produced real Markdown headings but did not preserve their nesting.View Result →

It put subsections at the same level as their parent sections and turned two main sections into lower-level headings.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

ReductoFailed1 of 1 test case failedFlattened the heading structure in the Markdown output.View Result →

The file contains one "#", 26 "##" headings, and no deeper levels; the document title is plain paragraph text. The two visible parent/child pairs were emitted as peers, so the nesting expected for styled headings and subheadings was not preserved.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Upstage AIFailed1 of 1 test case failedReturned every heading of the three-page procedure at a single Markdown level, so the source's three heading sizes collapsed into one and no nesting survived.View Result →

Three headings, including Refrigerated storage, came back as plain lines with no heading marker.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 2
Adobe PDF Extract APIPassed1 of 1 test case passedPreserved the styled heading hierarchy in the Markdown produced for this test.View Result →

Its structured output supplied three heading levels, retaining the source’s order and nesting.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

FirecrawlPassed1 of 1 test case passedPasses this styled-heading conversion case.View Result →

The returned Markdown keeps the three heading tiers as H1, H2 and H3 in the same order and nesting, and the letterhead stays plain text.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (4): Mistral AI, PDF Vector, Tensorlake, Unstructured.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
1 public · 0 withheld
Capabilities exercised here
Heading & Section Structure

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

A document with styled headings and subheadings in Converting a complex PDF into clean Markdown with a hosted API | AI Demos