The answer was never in the schema
A question asks about a fact that was not included in the schema, and any document-based answer must be explicitly attributed.
What this scenario means
This scenario tests whether the system recognises that the requested fact was not part of the extracted schema. A correct response may either say the fact is not in the extracted data or answer from the document with clear attribution. It must not present a document-based answer as extracted structured data, and it must not invent a value.
What we evaluate
- Whether it says the requested fact is not in the extracted data.
- Whether any answer taken from the document is explicitly attributed to the document.
- Whether a document-based answer is avoided when it would be presented as extracted structured data.
- Whether it avoids inventing a value.
Capabilities this scenario exercises
A scenario may exercise one or more capabilities.
Querying
Operates over the extracted records — filters, totals, answers questions — and is clear when an answer didn't come from them.
Benchmarks that use this scenario
A scenario has global identity and may be reused across benchmarks.
Structured Document Extraction
Which document extraction platform turns a user-defined schema into correct structured records — across layouts, scans and repeated sets — and is honest about what it could not find?