Aggregate across documents
A request asks for a total over the extracted records from all source documents.
What this scenario means
This scenario tests whether the tool can aggregate one numeric field across the extracted records from every source document and return the correct total only when every document contributes. The answer is judged from the extracted records the tool produced, not from the original documents.
What we evaluate
- Whether it returns a total built from the extracted records from all source documents.
- Whether the total depends on every document contributing to the aggregate.
- Whether it avoids including any value that was not among the extracted records.
- Whether it returns a refusal instead of the total.
Capabilities this scenario exercises
A scenario may exercise one or more capabilities.
Querying
Operates over the extracted records — filters, totals, answers questions — and is clear when an answer didn't come from them.
Benchmarks that use this scenario
A scenario has global identity and may be reused across benchmarks.
Structured Document Extraction
Which document extraction platform turns a user-defined schema into correct structured records — across layouts, scans and repeated sets — and is honest about what it could not find?