Scenario in benchmark · Converting a complex PDF into clean Markdown with a hosted API · Version 1
A document with figures and captions
A document contains a figure with its caption attached.
What we check
- Whether the figure is preserved rather than omitted or replaced by surrounding prose.
- Whether the figure remains positioned in the output at the point where it belongs in the document flow.
- Whether the caption stays attached to the figure and remains associated with it.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
FirecrawlFailed1 of 1 test case failedDid not preserve the figure with its caption in the converted document.View Result →
It kept the survey-plan caption in position but returned no image.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Landing AIFailed1 of 1 test case failedKept the site map's caption word for word at the figure's place in the returned Markdown, but did not keep the picture itself.View Result →
The figure holds only a generated description, with no image or link to one, and the response has no field that could carry an image.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
LlamaParseFailed1 of 1 test case failedReplaced the figure with prose rather than preserving it in the default Markdown output.View Result →
Its caption remained in place, but the image was neither embedded nor linked.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Nutrient.ioFailed1 of 1 test case failedNutrient.io's default Markdown conversion kept the figure's caption word for word and in its place, but dropped the figure itself: the returned Markdown has no image, link or placeholder where the survey-transect plan stood, and no image file came back with it.View Result →
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
PDF VectorFailed1 of 1 test case failedFailed to keep the figure with its caption in place.View Result →
The caption stayed in the right spot, but the figure itself was missing from the Markdown and the response, and the run card saved no asset files.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
ReductoFailed1 of 1 test case failedDid not preserve the figure as an image in Markdown.View Result →
It kept the caption in place but returned neither an embedded image nor an image link.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Upstage AIFailed1 of 1 test case failedRetained the figure’s caption but not the image itself.View Result →
A placeholder occupied the right position, but it did not link to an image.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Mistral AIPassed1 of 1 test case passedPreserved the figure at its source-page position and kept the caption directly below it.View Result →
The Markdown places `` between the two surrounding paragraphs, with the Figure 1 caption immediately beneath it.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (5): Adobe PDF Extract API, Extend AI, GPT-5.6 terra, Tensorlake, Unstructured.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 1 public · 0 withheld
- Capabilities exercised here
- Figures & Charts
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →