Scenario in benchmark · Structured Document Extraction · Version 1
Trace a value to its location
A returned value must be linked to the place in the document where it came from.
What we check
- Whether the tool points to the document location that matches the returned value.
- Whether the location is attached to the value it actually returned, not to a different one.
- Whether the tool gives a source location rather than only repeating the value or naming the document.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
FutureSmart Document IntelligenceFailed1 of 1 test case failedFailed to trace the subtotal to its document location.View Result →
The subtotal was on page 2, but the saved result had no location field, the chat only named a chunk, and the magnifier highlighted text in the extracted-text panel rather than a returned pointer.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Extend AIPassed1 of 1 test case passedTraced the returned value to the correct document location.View Result →
It returned 6103 with a citation to page 2 on "Subtotal" beside £6,103.00. The PDF shows "Subtotal" once on page 2 and not on page 1.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
LlamaParsePassed1 of 1 test case passedTraced the value to the right place in the document.View Result →
The value appears once in the two-page invoice; the citation names page 2 and the viewer box lands on the value.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
NanonetsPassed1 of 1 test case passedTraced the returned subtotal to page 2, where GBP 6,103.00 is highlighted in the totals block.View Result →
The exported page number was 2, and the preview highlight sat on the returned value rather than the neighboring totals.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
ReductoPassed1 of 1 test case passedGave a precise source location for the extracted value.View Result →
It returned the £6,103.00 subtotal with a citation to the amount itself on page 2.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (5): Datalab, Docsumo, Landing AI, Retab, Unstract.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Source Grounding
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →