Scenario in benchmark · Converting a complex PDF into clean Markdown with an open-source library · Version 1
A document containing mathematical equations
A document containing mathematical equations is handled as mathematics, not flattened into ordinary prose.
What we check
- Whether the equations remain represented as math rather than being flattened into ordinary prose.
- Whether the mathematical structure is preserved well enough to stay readable and recoverable.
- Whether any limitation is stated honestly through a clear fallback instead of silent corruption.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
LiteParseFailed1 of 1 test case failedTurned the note's equations into scrambled text: its Markdown has no math markup and no placeholder, each display equation is split across lines in the wrong order, the ω of equation (4)'s ω/c factor is missing, and the top of equation (6) became a heading.View Result →
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
MarkItDownFailed1 of 1 test case failedCorrupted all six equations rather than preserving them as math or providing an honest fallback.View Result →
Fractions, exponents and subscripts became disordered fragments, while the surrounding prose remained accurate.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
PyMuPDF4LLMFailed1 of 1 test case failedDid not preserve the document’s equations in Markdown.View Result →
It omitted all six display equations without providing math markup, an image or a placeholder.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
DoclingPassed1 of 1 test case passedPassed this equation-handling case.View Result →
It converted the three-page PDF in the default pipeline and replaced all six display equations with explicit fallback markers in the right places instead of garbled prose. The inline mathematical symbols were moved out of some sentences, and no LaTeX or MathML appears in the output.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (6): doc2mark, GPT-5.6 terra, Marker, MinerU, olmOCR, PaddleOCR-VL.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 1 public · 0 withheld
- Capabilities exercised here
- Equations & Mathematical Notation
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →