A field the tool can only get right from examples
The tool learns an implicit rule from labeled examples and applies it to a new unseen document.
What this scenario means
This scenario tests whether a product can infer a rule from example documents paired with their expected output, rather than only following a rule written into the schema or prompt. A good agent uses those examples to learn the hidden convention, then applies that same convention to a fresh document it has not seen before. The hard part is generalisation: the rule is not stated directly, so success depends on learning from evidence, not on configuration.
What we evaluate
- Whether the tool can infer a rule from labeled examples rather than from an explicit instruction.
- Whether it applies the learned rule to a document it has not seen before.
- Whether it follows the convention shown in the examples when producing the new result.
- Whether it generalises consistently without needing the rule to be stated in the schema or prompt.
Capabilities this scenario exercises
A scenario may exercise one or more capabilities.
Training
Learns from labeled examples — documents paired with their expected output — and applies what it learned to a document it hasn't seen.
Benchmarks that use this scenario
A scenario has global identity and may be reused across benchmarks.
Structured Document Extraction
Which document extraction platform turns a user-defined schema into correct structured records — across layouts, scans and repeated sets — and is honest about what it could not find?