Scenario in benchmark · Structured Document Extraction · Version 1
A repeated set of records
A document presents repeated records that must all be extracted with the right values and record association.
What we check
- Whether every genuine record in the repeated set is returned.
- Whether each value stays attached to the correct record.
- Whether no extra records are invented and no genuine record is omitted.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
DocsumoPassed1 of 1 test case passedExtracted the invoice’s three line items with the right values attached to each item.View Result →
It excluded the subtotal, VAT and total rows from the extracted set.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
FutureSmart Document IntelligencePassed1 of 1 test case passedPassed this repeated-set case.View Result →
It returned three line items with the right descriptions, quantities, unit prices, and amounts in the right rows; item code was requested but missing from every record.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Landing AIPassed1 of 1 test case passedExtracted the repeated set of records in this run.View Result →
The invoice table on page 3 returned three line items with the correct values and pairing; the delivery note says there were three lines and six units, and the schema asked for description and amount only.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
LlamaParsePassed1 of 1 test case passedHandled the repeated set correctly.View Result →
It returned all three records with the right values, and the per-value citations stayed in separate rows, so nothing was bound to the wrong record.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
NanonetsPassed1 of 1 test case passedKept the repeated invoice items separate and preserved their associated values.View Result →
The output contained three objects for the source table's three rows.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
ReductoPassed1 of 1 test case passedReturned all three line items from the invoice's goods table, with all fifteen values matching the table and each value cited from its own item's row.View Result →
It did not return the Subtotal, VAT or Total due rows as items.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Nested & Repeated Fields
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →