Scenario in benchmark · Structured Document Extraction · Version 1

Aggregate across documents

A request asks for a total over the extracted records from all source documents.

Capability: Querying1 pinned test case per toolNeutral global definition →

What we check

  1. Whether it returns a total built from the extracted records from all source documents.
  2. Whether the total depends on every document contributing to the aggregate.
  3. Whether it avoids including any value that was not among the extracted records.
  4. Whether it returns a refusal instead of the total.

Specific test-case inputs and grading live on the related test-case pages.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 3
Extend AIFailed1 of 1 test case failedReturned £7,838.04 instead of the required £9,592.44 when asked to total three invoices.View Result →

The shortfall equals one invoice’s £1,754.40 contribution.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Landing AIFailed1 of 1 test case failedFails to total across uploaded documents.View Result →

Chat answered with one file’s own amount or refused, and Extract returned each invoice’s own total in the cross-document field instead of one combined sum.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

LlamaParseFailed1 of 1 test case failedDid not produce the required combined total of 474.93 across three receipts in Extract on the Free plan.View Result →

It filled the requested cross-document sum with individual receipt totals instead.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 3
DocsumoPassed1 of 1 test case passedCorrectly totaled values across multiple documents in this case.View Result →

It returned 474.93 for the three receipts and showed each document’s value separately.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

FutureSmart Document IntelligencePassed1 of 1 test case passedAggregated across the three named invoices and returned the exact sum, £10,327.64.View Result →

The Coppergate invoice's £1,982.00 total matched its line items, and the answer itemized the three document totals without any document missing or extra value added.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

NanonetsPassed1 of 1 test case passedCombined the amounts due from all three documents into the expected total of 474.93.View Result →

The calculation left out no document and added no value from outside the set.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (4): Datalab, Reducto, Retab, Unstract.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Querying

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

Aggregate across documents in Structured Document Extraction | AI Demos