Scenario in benchmark · Structured Document Extraction · Version 1
A field the tool can only get right from examples
The tool learns an implicit rule from labeled examples and applies it to a new unseen document.
What we check
- Whether the tool can infer a rule from labeled examples rather than from an explicit instruction.
- Whether it applies the learned rule to a document it has not seen before.
- Whether it follows the convention shown in the examples when producing the new result.
- Whether it generalises consistently without needing the rule to be stated in the schema or prompt.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
FutureSmart Document IntelligenceDifferent test setDid not show learning from examples on this field.View Result →
It returned the correct label, business, for the unseen invoice in both the seven-document collection and the separate control collection, and it offers no way to supply labeled examples or train from them.
Published on a different test set, so its counts are not compared here.
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (9): Datalab, Docsumo, Extend AI, Landing AI, LlamaParse, Nanonets, Reducto, Retab, Unstract.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Training
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 pins a scenario rubric for this scenario. Benchmark methodology →