Scenario in benchmark · Converting a complex PDF into clean Markdown with a hosted API · Version 1
A simple, clearly formatted table
A simple, clearly formatted table is provided for processing.
What we check
- Whether the rows and columns remain aligned after processing.
- Whether each value stays under the correct header.
- Whether the output still reads as a table rather than a flattened or split block of text.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
Nutrient.ioFailed1 of 1 test case failedFailed to keep the simple table intact in the default Build API run.View Result →
The output kept all values, row count, and row order, but merged the method and turnaround headers and cells.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Adobe PDF Extract APIPassed1 of 1 test case passedPreserved the simple table’s structure and values in the Markdown produced for this test.View Result →
All four headings and 32 data cells matched the source.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
FirecrawlPassed1 of 1 test case passedPreserved the structure of the simple table in this test.View Result →
It returned all four headers and eight rows, with the values under the correct headers.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
GPT-5.6 terraPassed1 of 1 test case passedPreserved the simple table in Markdown.View Result →
It kept the four headers and eight rows in order, with each code, turnaround and charge matching the source; the surrounding text stayed word-exact apart from the repeated footer.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
LlamaParsePassed1 of 1 test case passedReturned the page-1 charges table intact: the four headers in order, all eight rows and every value under the right column.View Result →
The cells most likely to drift held: 'Metals suite, eleven elements' stayed one cell, and '10 days' stayed out of the charge column. The table came back as an HTML table inside the Markdown.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Mistral AIPassed1 of 1 test case passedPreserved a simple, clearly formatted table: the returned Markdown kept the four headers in order, retained eight data rows, and matched all 32 data cells against the source.View Result →
The table stayed a single pipe table, and the comma inside "Metals suite, eleven elements" did not split the cell.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
ReductoPassed1 of 1 test case passedKept the simple table as a table in Markdown, with the same headers, rows, columns, and values.View Result →
In the checked run, the table stayed in the right place on page 1, and pages 2 and 3 had no table. The table used HTML table markup rather than pipe syntax.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (6): Extend AI, Landing AI, PDF Vector, Tensorlake, Unstructured, Upstage AI.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 1 public · 0 withheld
- Capabilities exercised here
- Table Extraction
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →