Scenario in benchmark · Structured Document Extraction · Version 1

The value needs a supplied definition

When several plausible values fit a field name, the schema-supplied definition decides which value counts.

Capability: Field Extraction1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the returned field value follows the supplied definition when more than one plausible value is present.
  2. Whether the tool resolves the ambiguity from schema context rather than from the most visible candidate.
  3. Whether the extracted record contains the schema-selected value and not another plausible value.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Passed · 6
DocsumoPassed1 of 1 test case passedSelected the value specified by the supplied definition.View Result →

It returned the shipment reference rather than the more prominent invoice number.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Extend AIPassed1 of 1 test case passedSelected the consignment reference required by the supplied definition, not the more prominent invoice number.View Result →

It cited the consignment line on page 1.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

FutureSmart Document IntelligencePassed1 of 1 test case passedPassed this case by returning the consignment reference instead of the more prominent invoice number.View Result →

The saved instruction explicitly excluded the invoice number, the output was CFS-CN-40821, and the invoice number appeared repeatedly in the document.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Landing AIPassed1 of 1 test case passedReturned the value selected by the supplied definition instead of the more prominent alternative.View Result →

The invoice number CFS-7710 was the more prominent candidate, and the returned value, CFS-CN-40821, matched the consignment reference.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

NanonetsPassed1 of 1 test case passedChose the consignment reference, not the invoice number, for this field.View Result →

The result row and exported JSON both show CFS-CN-40821, matching the supplied definition.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

ReductoPassed1 of 1 test case passedFollowed the supplied definition when two references could fit the field name.View Result →

It returned the consignment reference, not the more prominent invoice number.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (4): Datalab, LlamaParse, Retab, Unstract.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Field Extraction

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

The value needs a supplied definition in Structured Document Extraction | AI Demos