Scenario in benchmark · Converting a complex PDF into clean Markdown with a hosted API · Version 1

A document containing mathematical equations

A document containing mathematical equations is handled as mathematics, not flattened into ordinary prose.

What we check

  1. Whether the equations remain represented as math rather than being flattened into ordinary prose.
  2. Whether the mathematical structure is preserved well enough to stay readable and recoverable.
  3. Whether any limitation is stated honestly through a clear fallback instead of silent corruption.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 2
Extend AIFailed1 of 1 test case failedFailed to keep equations as math markup or an honest fallback in Markdown.View Result →

The returned Markdown turned equations (1) to (4) into garbled prose and never showed math markup or a stated fallback, even though the source equations were available in the PDF.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

ReductoFailed1 of 1 test case failedFailed a document with mathematical equations: the returned Markdown contained no math markup and no fallback, and the equations were flattened into prose with missing symbols and substituted characters.View Result →

This is the case: the source page shows set mathematics, while the output preserves only garbled lines instead of LaTeX, MathML, or an explicit fallback.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 3
Adobe PDF Extract APIPassed1 of 1 test case passedPreserved the equations in the three-page PDF as linked figure images in the generated Markdown.View Result →

The page shows each equation in place with its number beneath it, and the figure images keep fractions, superscripts, subscripts, and absolute-value bars intact; there are no LaTeX, MathML, or $ delimiters.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

GPT-5.6 terraPassed1 of 1 test case passedKept the document's equations as math markup rather than flattening them into prose.View Result →

The source note had six numbered equations plus one unnumbered relation, and the returned Markdown carried all seven as LaTeX with numbering preserved; the title, byline, sections, body prose, and references also carried over.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Mistral AIPassed1 of 1 test case passedKept the mathematical equations as LaTeX math in the Markdown for this PDF.View Result →

The returned Markdown preserved all six display equations as math blocks with tags, kept inline symbols inside math delimiters, and carried through the page-break text and running heads. The page-by-page comparison sheets matched the source pages shown there.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (8): Firecrawl, Landing AI, LlamaParse, Nutrient.io, PDF Vector, Tensorlake, Unstructured, Upstage AI.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
1 public · 0 withheld
Capabilities exercised here
Equations & Mathematical Notation

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

A document containing mathematical equations in Converting a complex PDF into clean Markdown with a hosted API | AI Demos