Scenario in benchmark · Structured Document Extraction · Version 1
The field is absent
The requested fact is absent and not implied by the document.
What we check
- Whether the output gives an honest null or absence response for the missing fact.
- Whether the output avoids inventing a value for the absent fact.
- Whether any invented value is left without a pointer or other grounding evidence.
Specific test-case inputs and grading live on the related test-case pages.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
DocsumoPassed1 of 1 test case passedPassed this scenario: it returned an honest null for an absent field instead of inventing a value, and the empty field had no source pointer.View Result →
In the same run, the receipt number, subtotal, tax, and total stayed correct.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Extend AIPassed1 of 1 test case passedPasses this absent-field case: it returned null for the added field and no grounding to an invented value.View Result →
The receipt contains no loyalty or membership data, and the response carries an empty citations array.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
FutureSmart Document IntelligencePassed1 of 1 test case passedReturned an honest null for the absent field.View Result →
The structured data row shows the loyalty cell empty while sibling fields are filled, and the document details output carries only document-level fields; the source text has no loyalty or member mention.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Landing AIPassed1 of 1 test case passedPassed this scenario by returning an honest null for the absent field.View Result →
The metadata showed no source range for that field. The receipt number was returned as RHS-70234, with a source range shown in the metadata.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
LlamaParsePassed1 of 1 test case passedPassed this absent-field case by leaving the missing loyalty field empty and ungrounded.View Result →
The receipt has no loyalty wording, the extractor returned an empty value for that field, and the job record gave that field no citation metadata while the present receipt number had a citation.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
NanonetsPassed1 of 1 test case passedLeft the absent loyalty membership number empty without inventing a source.View Result →
It explicitly reported that the document did not contain the number.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
ReductoPassed1 of 1 test case passedDid not invent a value for a field the receipt does not contain.View Result →
The receipt contains no loyalty, membership, or account reference, and the run returned a real null for the missing field with no citation, while the control field came back correct and cited.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Field Extraction, Source Grounding
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →