Scenario in benchmark · Structured Document Extraction · Version 1

Corrected data goes downstream

After a record is corrected, a later downstream read shows the corrected value.

Capability: Review and Correction1 pinned test case per toolNeutral global definition →

What we check

  1. Whether a later read returns the corrected value rather than the original extraction.
  2. Whether the downstream read reflects the updated record after correction.
  3. Whether the tool continues to show the correction when the record is read again later.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 2
FutureSmart Document IntelligenceFailed1 of 1 test case failedDid not keep corrected data downstream.View Result →

The export path existed, but it carried only the original extraction, and the receipt row in the downloaded CSV still held RHS-70234.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

LlamaParseFailed1 of 1 test case failedDid not carry the corrected receipt_number RHS-99999 downstream on the Extract screen.View Result →

On the Free plan, the agentic four-field setup stayed read-only, and the downloaded JSON and opened export still showed receipt_number RHS-70234 with subtotal, tax, and total_due.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 3
DocsumoPassed1 of 1 test case passedKept the correction in the delivery path for this case.View Result →

The spreadsheet and JSON export both show the corrected receipt number and matching money fields, while the source receipt shows the earlier number. One observation notes the spreadsheet stores total_due as a float.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Extend AIPassed1 of 1 test case passedDelivered the corrected receipt number through its API.View Result →

The subtotal, tax and total due stayed at $85.00, $5.10 and $90.10.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

NanonetsPassed1 of 1 test case passedPasses this scenario.View Result →

In the tested case, the corrected receipt number appears in the product’s delivery path: the result grid shows RHS-99999, the Download menu is open on that grid, and the exported JSON carries RHS-99999 with the other receipt values.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (5): Datalab, Landing AI, Reducto, Retab, Unstract.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Review and Correction

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

Corrected data goes downstream in Structured Document Extraction | AI Demos