Scenario in benchmark · Converting a complex PDF into clean Markdown with a hosted API · Version 1

A document with a data chart

A document includes a data chart that must be converted as part of the page.

Capability: Figures & Charts1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the chart remains present as a visual asset in the converted output.
  2. Whether the chart stays associated with its original position and its title or caption.
  3. Whether any extra textual chart details are treated as optional enrichment, not as the minimum bar.
  4. Whether the visual chart is not replaced by prose alone.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 6
FirecrawlFailed1 of 1 test case failedDid not preserve the data chart in this PDF-to-Markdown conversion.View Result →

Its default output omitted both the chart image and its title.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Landing AIFailed1 of 1 test case failedDid not preserve the data chart as an image.View Result →

It retained the chart’s title in place and supplied a description and a table of approximate values.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

PDF VectorFailed1 of 1 test case failedFailed to preserve the chart as an image in the graded run.View Result →

The default output returned Markdown only and left the chart, its title, and its axis label out of the page-3 slot; separate pro and max runs brought back the chart text, and max added HTML figure markup, but neither returned an image.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

ReductoFailed1 of 1 test case failedOmitted the data chart’s image and title from the Markdown.View Result →

A prose description appeared where the chart had been.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

UnstructuredFailed1 of 1 test case failedUnstructured's default conversion did not preserve the data chart as an image in Markdown.View Result →

It supplied a text description, while the plan on page 2 retained its image.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Upstage AIFailed1 of 1 test case failedFailed this PDF-chart scenario: the page-3 chart was not preserved as an image with its title.View Result →

The Markdown shows a placeholder image, 'Chart Type: bar', and a wrong value table; the surrounding prose and page-2 Figure 1 caption remain.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 2
Adobe PDF Extract APIPassed1 of 1 test case passedPreserved the chart as an image with its title.View Result →

It did not add the chart's data as text.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Mistral AIPassed1 of 1 test case passedKept the page-3 bar chart as an image in its original place, with the chart's title as bold text directly above it.View Result →

It returned no table and none of the chart's values: the chart is kept as a picture, not turned into data.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (5): Extend AI, GPT-5.6 terra, LlamaParse, Nutrient.io, Tensorlake.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
1 public · 0 withheld
Capabilities exercised here
Figures & Charts

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

A document with a data chart in Converting a complex PDF into clean Markdown with a hosted API | AI Demos