Scenario in benchmark · Structured Document Extraction · Version 1

The set contains a non-record

A repeated set contains an explicitly identified non-record that resembles a member but must be excluded.

Capability: Nested & Repeated Fields1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the tool excludes the explicitly marked non-record from the extracted set.
  2. Whether the tool returns only genuine members of the repeated set.
  3. Whether the tool avoids dropping a real member while removing the non-record.
  4. Whether set membership stays correct even when the non-record matches the surrounding row pattern.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Passed · 5
DocsumoPassed1 of 1 test case passedExcluded the invoice’s total row without losing a genuine line item.View Result →

It returned the five invoice items, preserving each row’s number, description, quantity, unit price and amount.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Extend AIPassed1 of 1 test case passedExtracted the genuine invoice items without including the total row.View Result →

It returned five items from five genuine source rows, with none missing or substituted.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

FutureSmart Document IntelligencePassed1 of 1 test case passedPassed this non-member scenario.View Result →

It kept the five genuine rows, omitted the Total row, showed the same five items on screen, and treated the 1,982.00 GBP total as document-level information.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Landing AIPassed1 of 1 test case passedLeft the non-record out of the extracted set in the tested invoice.View Result →

The output contained five members, matching the five numbered rows, and it did not include the separate Total row or its 1,982.00 amount. The schema description shown in the capture was neutral.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

NanonetsPassed1 of 1 test case passedPassed this lookalike-row scenario.View Result →

It returned five genuine line items and left out the lookalike Total row; the input table shows that row in the same layout, and the output has no 'Total' entry or 1,982 amount.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (5): Datalab, LlamaParse, Reducto, Retab, Unstract.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Nested & Repeated Fields

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

The set contains a non-record in Structured Document Extraction | AI Demos