Scenario in benchmark · Structured Document Extraction · Version 1
Aggregate across documents
A request asks for a total over the extracted records from all source documents.
What we check
- Whether it returns a total built from the extracted records from all source documents.
- Whether the total depends on every document contributing to the aggregate.
- Whether it avoids including any value that was not among the extracted records.
- Whether it returns a refusal instead of the total.
Specific test-case inputs and grading live on the related test-case pages.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
Extend AIFailed1 of 1 test case failedReturned £7,838.04 instead of the required £9,592.44 when asked to total three invoices.View Result →
The shortfall equals one invoice’s £1,754.40 contribution.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Landing AIFailed1 of 1 test case failedFails to total across uploaded documents.View Result →
Chat answered with one file’s own amount or refused, and Extract returned each invoice’s own total in the cross-document field instead of one combined sum.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
LlamaParseFailed1 of 1 test case failedDid not produce the required combined total of 474.93 across three receipts in Extract on the Free plan.View Result →
It filled the requested cross-document sum with individual receipt totals instead.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
DocsumoPassed1 of 1 test case passedCorrectly totaled values across multiple documents in this case.View Result →
It returned 474.93 for the three receipts and showed each document’s value separately.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
FutureSmart Document IntelligencePassed1 of 1 test case passedAggregated across the three named invoices and returned the exact sum, £10,327.64.View Result →
The Coppergate invoice's £1,982.00 total matched its line items, and the answer itemized the three document totals without any document missing or extra value added.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
NanonetsPassed1 of 1 test case passedCombined the amounts due from all three documents into the expected total of 474.93.View Result →
The calculation left out no document and added no value from outside the set.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Querying
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →