Scenario in benchmark · Structured Document Extraction · Version 1

The set is legitimately empty

A document explicitly indicates that a repeated set contains no items.

Capability: Nested & Repeated Fields1 pinned test case per toolNeutral global definition →

What we check

  1. Whether it returns an empty set or empty array for the repeated field.
  2. Whether it avoids inventing any member when the document says there are none.
  3. Whether it preserves the schema-required shape for the field.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Passed · 6
DocsumoPassed1 of 1 test case passedPassed this empty-set extraction case.View Result →

The page said no damage or exceptions were reported, and the extracted table came back with zero rows in both the saved JSON and the Results panel.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Extend AIPassed1 of 1 test case passedReturned an empty set for the legitimately empty document.View Result →

The extracted set of damage or exception entries is empty, and the source references are empty too. Page 3 says no damage or exceptions were reported.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

FutureSmart Document IntelligencePassed1 of 1 test case passedPassed this empty-set extraction case.View Result →

It returned damage_exceptions as an empty array, and the document says no damage or exceptions were reported; the overview card omits that field.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Landing AIPassed1 of 1 test case passedReturned an empty set when the document said there were none this period.View Result →

In the checked PDF page, the source text said no damage or exceptions were reported, and the export showed an empty array. The schema pane showed the field as a required array, and the output did not turn the marker text into a member.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

LlamaParsePassed1 of 1 test case passedPassed this empty-set extraction case.View Result →

It returned an empty required array for the damage-exceptions field, with the source page saying nothing was reported and no invented member or marker text added.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

NanonetsPassed1 of 1 test case passedReturned the empty set the document called for.View Result →

The input says no damage or exceptions were reported, and the output is an empty array; the capture shows the same field as List [0] with no rows.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (4): Datalab, Reducto, Retab, Unstract.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Nested & Repeated Fields

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

The set is legitimately empty in Structured Document Extraction | AI Demos