Scenario in benchmark · Structured Document Extraction · Version 1

The value must be derived

A requested value is not stated directly, but can be inferred or computed from values that are present.

Capability: Field Extraction1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the tool derives a requested field from values already present in the document.
  2. Whether it returns the inferred value when the schema asks for something not stated directly.
  3. Whether it distinguishes a computed answer from a value copied verbatim from the page.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Passed · 7
DocsumoPassed1 of 1 test case passedCalculated a fuel cost that was not printed on the invoice.View Result →

It returned £81.00 from 450 miles at £0.18 per mile and preserved the decimal fuel rate.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Extend AIPassed1 of 1 test case passedDerived a fuel cost that was not stated in the invoice.View Result →

It returned £81.00, the required result of multiplying 450 miles by £0.18 per mile.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

FutureSmart Document IntelligencePassed1 of 1 test case passedDerived the missing fuel cost for this invoice.View Result →

The invoice shows 450 miles and £0.18 per mile, and the product returned 81 in both the JSON output and the overview card.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Landing AIPassed1 of 1 test case passedDerived the requested value in this scenario.View Result →

The invoice shows DISTANCE 450 miles and FUEL RATE £0.18 per mile; the JSON returns fuel_cost as GBP 81.00, and the document text does not contain 81.00. The playground screen shows the same result beside a one-field schema.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

LlamaParsePassed1 of 1 test case passedPassed this derived-value task.View Result →

It read the two page-1 operands and returned £81.00; the derived field had no citation, and the source contains no 81.00.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

NanonetsPassed1 of 1 test case passedDerived the requested value from the two present inputs and returned 81 for total_fuel_cost.View Result →

The two operand rows carried source-link icons in the screenshot and the total row did not, matching the JSON output and showing the value as a computed field.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

ReductoPassed1 of 1 test case passedCalculated a fuel cost of £81.00 from the invoice's 450 miles and £0.18 per mile.View Result →

The requested total was not printed anywhere in the document.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (3): Datalab, Retab, Unstract.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Field Extraction

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

The value must be derived in Structured Document Extraction | AI Demos