Structured Document Extraction
This benchmark evaluates document extraction platforms that take a user-defined schema and return structured records.
Benchmark overview
What is included and excluded
Evaluates whether a document extraction platform can turn a user-defined schema into correct structured records across layouts, scans and repeated sets, while staying honest about what it could not find.
A buyer learns how far the platform can go in turning documents into usable structured output, and whether it clearly separates extracted facts from missing ones or values found elsewhere in the document.
In scope
- User-defined schema to structured records from many documents
- Scalar fields and repeated or nested records
- Digital, scanned and mixed documents
- Training from labelled examples
- Correcting returned data
- Working with extracted records: filtering, totalling and asking questions
Out of scope
- Document classification
- Packet splitting
- Routing document types to different schemas
- Transcription quality
- Non-English documents
Participating tools
How each tool did on the 19 scenarios in this benchmark. Each cell is one scenario, in the same order for every tool and grouped by capability; Pass and Fail come from published Results.
| Tool | Scenario results · 19 scenarios in 7 capabilities | Passed | Failed | Explore |
|---|---|---|---|---|
| FutureSmart Document Intelligence | 15of 19 graded | 4 | View tool → | |
| Nanonets | 16of 17 graded | 1 | View tool → | |
| LlamaParse | 11of 16 graded | 5 | View tool → | |
| Extend AI | 12of 15 graded | 3 | View tool → | |
| Landing AI | 12of 15 graded | 3 | View tool → | |
| Docsumo | 14of 14 graded | 0 | View tool → | |
| Reducto | 7of 8 graded | 1 | View tool → | |
| Datalab | 2of 2 graded | 0 | View tool → | |
| 2 more toolsin the roster | No published Results yet for Retab · Unstract.Each tool gets its own row here once its first Result is published. | |||
Capabilities & scenarios
19 scenarios grouped by 7 capabilities. Open a group to explore its scenarios in this benchmark.
Field Extraction5 scenarios
Returns the scalar fields the schema asks for — present, interpreted, derived, absent or differently laid out — correctly typed
Capability in this benchmark → · Global definition →
- The value is directly available6 tools with published Results
- The value needs a supplied definition6 tools with published Results
- The value must be derived7 tools with published Results
- The same field across layouts5 tools with published Results
- The field is absent7 tools with published Results
Nested & Repeated Fields4 scenarios
Returns repeated sets of records completely, with each value under the right record: none dropped, none invented, none cross-bound
Capability in this benchmark → · Global definition →
- A repeated set of records6 tools with published Results
- The set continues across a page break5 tools with published Results
- The set contains a non-record5 tools with published Results
- The set is legitimately empty6 tools with published Results
Training1 scenario
Learns from labeled examples — documents paired with their expected output — and applies what it learned to a document it hasn't seen
Capability in this benchmark → · Global definition →
- A field the tool can only get right from examples1 tool with a published Result
OCR2 scenarios
The same schema returns the same values when the document is an image, and a mixed document is handled page by page
Capability in this benchmark → · Global definition →
- A cleanly scanned document5 tools with published Results
- A document mixing digital and scanned pages7 tools with published Results
Source Grounding2 scenarios
Points to where a returned value came from — and attaches no pointer to a value that wasn't there
Capability in this benchmark → · Global definition →
- Trace a value to its location5 tools with published Results
- The field is absent7 tools with published Results
Review and Correction3 scenarios
A human fix reaches the stored record and the systems downstream of it
Capability in this benchmark → · Global definition →
- Correct a field value3 tools with published Results
- Repair a record5 tools with published Results
- Corrected data goes downstream5 tools with published Results
Querying3 scenarios
Operates over the extracted records — filters, totals, answers questions — and is clear when an answer didn't come from them
Capability in this benchmark → · Global definition →
- Filter records across documents4 tools with published Results
- Aggregate across documents6 tools with published Results
- The answer was never in the schema6 tools with published Results
Results overview
Current published evidence in this benchmark.
Publication availability is separate from test coverage and unpublished research progress.
How the benchmark works
A public summary of the evaluation method. The same defined scope and evidence standard apply to every tool assessed under this version.
Unsupported capabilities
A capability a tool doesn't support scores 0, and the 0 stays in the denominator. No tool ranks higher for having less product.
Resources and fixtures
The registered material and systems that create a consistent test environment for this benchmark.
Invoice, receipt and bank statement test documents
A fixed round-1 fixture corpus of 19 fictional PDFs for Structured Document Extraction (R5 v1).