Scenario in benchmark · Structured Document Extraction · Version 1

The field is absent

The requested fact is absent and not implied by the document.

Capability: Field Extraction, Source Grounding1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the output gives an honest null or absence response for the missing fact.
  2. Whether the output avoids inventing a value for the absent fact.
  3. Whether any invented value is left without a pointer or other grounding evidence.

Specific test-case inputs and grading live on the related test-case pages.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Passed · 7
DocsumoPassed1 of 1 test case passedPassed this scenario: it returned an honest null for an absent field instead of inventing a value, and the empty field had no source pointer.View Result →

In the same run, the receipt number, subtotal, tax, and total stayed correct.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Extend AIPassed1 of 1 test case passedPasses this absent-field case: it returned null for the added field and no grounding to an invented value.View Result →

The receipt contains no loyalty or membership data, and the response carries an empty citations array.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

FutureSmart Document IntelligencePassed1 of 1 test case passedReturned an honest null for the absent field.View Result →

The structured data row shows the loyalty cell empty while sibling fields are filled, and the document details output carries only document-level fields; the source text has no loyalty or member mention.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Landing AIPassed1 of 1 test case passedPassed this scenario by returning an honest null for the absent field.View Result →

The metadata showed no source range for that field. The receipt number was returned as RHS-70234, with a source range shown in the metadata.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

LlamaParsePassed1 of 1 test case passedPassed this absent-field case by leaving the missing loyalty field empty and ungrounded.View Result →

The receipt has no loyalty wording, the extractor returned an empty value for that field, and the job record gave that field no citation metadata while the present receipt number had a citation.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

NanonetsPassed1 of 1 test case passedLeft the absent loyalty membership number empty without inventing a source.View Result →

It explicitly reported that the document did not contain the number.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

ReductoPassed1 of 1 test case passedDid not invent a value for a field the receipt does not contain.View Result →

The receipt contains no loyalty, membership, or account reference, and the run returned a real null for the missing field with no citation, while the control field came back correct and cited.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (3): Datalab, Retab, Unstract.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Field Extraction, Source Grounding

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

The field is absent in Structured Document Extraction | AI Demos