Scenario in benchmark · Structured Document Extraction · Version 1
The value must be derived
A requested value is not stated directly, but can be inferred or computed from values that are present.
What we check
- Whether the tool derives a requested field from values already present in the document.
- Whether it returns the inferred value when the schema asks for something not stated directly.
- Whether it distinguishes a computed answer from a value copied verbatim from the page.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
DocsumoPassed1 of 1 test case passedCalculated a fuel cost that was not printed on the invoice.View Result →
It returned £81.00 from 450 miles at £0.18 per mile and preserved the decimal fuel rate.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Extend AIPassed1 of 1 test case passedDerived a fuel cost that was not stated in the invoice.View Result →
It returned £81.00, the required result of multiplying 450 miles by £0.18 per mile.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
FutureSmart Document IntelligencePassed1 of 1 test case passedDerived the missing fuel cost for this invoice.View Result →
The invoice shows 450 miles and £0.18 per mile, and the product returned 81 in both the JSON output and the overview card.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Landing AIPassed1 of 1 test case passedDerived the requested value in this scenario.View Result →
The invoice shows DISTANCE 450 miles and FUEL RATE £0.18 per mile; the JSON returns fuel_cost as GBP 81.00, and the document text does not contain 81.00. The playground screen shows the same result beside a one-field schema.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
LlamaParsePassed1 of 1 test case passedPassed this derived-value task.View Result →
It read the two page-1 operands and returned £81.00; the derived field had no citation, and the source contains no 81.00.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
NanonetsPassed1 of 1 test case passedDerived the requested value from the two present inputs and returned 81 for total_fuel_cost.View Result →
The two operand rows carried source-link icons in the screenshot and the total row did not, matching the JSON output and showing the value as a computed field.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
ReductoPassed1 of 1 test case passedCalculated a fuel cost of £81.00 from the invoice's 450 miles and £0.18 per mile.View Result →
The requested total was not printed anywhere in the document.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Field Extraction
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →