Scenario in benchmark · Structured Document Extraction · Version 1

The answer was never in the schema

A question asks about a fact that was not included in the schema, and any document-based answer must be explicitly attributed.

Capability: Querying1 pinned test case per toolNeutral global definition →

What we check

  1. Whether it says the requested fact is not in the extracted data.
  2. Whether any answer taken from the document is explicitly attributed to the document.
  3. Whether a document-based answer is avoided when it would be presented as extracted structured data.
  4. Whether it avoids inventing a value.

Specific test-case inputs and grading live on the related test-case pages.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 4
Extend AIFailed1 of 1 test case failedOffered no way to ask about a document fact outside the extraction schema.View Result →

The extraction returned the invoice reference and total due, without an answer about the printed due date.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

LlamaParseFailed1 of 1 test case failedOffered no place to ask about a fact printed on the receipt that the extraction schema left out.View Result →

Neither the Parse result screen and its menu nor the Extract results panel has a question or chat box, and Chat cannot start until an index exists.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

NanonetsFailed1 of 1 test case failedPresented a document answer as extracted data without distinguishing its source.View Result →

The first request correctly returned the receipt number, subtotal, tax and total due. The follow-up used the same extraction tool, placing the technician in a separate, similarly styled extraction block; the numbered source chip appeared only in the answer text below, which did not say the technician was outside the requested fields.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

ReductoFailed1 of 1 test case failedGives no way to ask a question about a document, so it cannot say whether an answer came from the extracted data or the document itself.View Result →

After one Parse request the technician printed on the receipt appeared among the parsed blocks; after one Extract request only the four schema fields came back.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 2
FutureSmart Document IntelligencePassed1 of 1 test case passedPassed this scenario.View Result →

It gave the IFSC code and then said it came from reading the document directly, not from a separate extracted metadata field.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Landing AIPassed1 of 1 test case passedPassed this scenario.View Result →

It answered the technician question with Renata Kowalski and showed clickable references to the receipt’s page text, while the receipt itself prints that name. Chat is experimental, works on one file at a time, and has no API, so the result is scoped to the web app.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (4): Datalab, Docsumo, Retab, Unstract.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Querying

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

The answer was never in the schema in Structured Document Extraction | AI Demos