Scenario in benchmark · Structured Document Extraction · Version 1
A document mixing digital and scanned pages
A document contains both digital text pages and scanned image-only pages.
What we check
- Whether the scanned page inside a mixed document is processed rather than silently skipped.
- Whether page-by-page handling keeps content from both digital text pages and scanned image-only pages.
- Whether the system treats the document as mixed, not as a fully digital file with no special handling needed.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
Landing AIFailed1 of 1 test case failedFailed this mixed digital-and-scanned PDF case.View Result →
All four fields came back, and the scanned-page values were exact, but the total due returned as "#1,246.80" even though the page text and rendering show "£1,246.80"; the run also carried both parse and extract ids.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
DatalabPassed1 of 1 test case passedReturned all five requested values from the three-page invoice, including the vehicle registration and the recipient's name that appear only on its scanned page.View Result →
Each value matched the document and was cited to the page it sits on, and the total due kept its pound sign.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
DocsumoPassed1 of 1 test case passedRead the scanned page inside the digital PDF rather than skipping it.View Result →
It returned all five requested values accurately, including the vehicle registration and delivery recipient found only on the scanned page.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Extend AIPassed1 of 1 test case passedReturned the requested values from a document mixing digital and scanned pages.View Result →
The decision-relevant finding is that the image-only page was processed rather than skipped: the scanned-page values and the digital-page values both came back correct in one extraction call.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
FutureSmart Document IntelligencePassed1 of 1 test case passedHandled the mixed-page document successfully.View Result →
It returned all five requested values exactly, including the two that existed only on the scanned page, and the three digital-page values matched unchanged.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
LlamaParsePassed1 of 1 test case passedExtracted the requested values from a document mixing digital and scanned pages.View Result →
It returned all five values correctly, including two found only on the scanned page.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
NanonetsPassed1 of 1 test case passedHandled the mixed digital-and-scanned document in this test.View Result →
It returned the five requested values, including the two that appear only in the scanned page image, and kept the digital-page values unchanged. The JSON output itself does not show per-value page provenance.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- OCR
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →