Scenario in benchmark · Structured Document Extraction · Version 1

Repair a record

A correction workflow repairs an extracted record by adding a needed member and removing an unwanted one.

Capability: Review and Correction1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the repaired set includes the intended member.
  2. Whether the repaired set no longer includes the unintended member.
  3. Whether the remaining members stay unchanged.
  4. Whether the corrected record persists after the repair is saved or passed on.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 2
FutureSmart Document IntelligenceFailed1 of 1 test case failedFails this repair task.View Result →

It could not add a new line item or find a delete option, and the read-back stayed at the same ten Blackwood items.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

LlamaParseFailed1 of 1 test case failedDid not let the user add or remove a member of an extracted set.View Result →

The saved result stayed at three line items: Site fencing remained, and Concrete mixer was absent after the edit attempts.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 3
DocsumoPassed1 of 1 test case passedRepaired the record by adding one line item and removing another.View Result →

After saving, reloading, and reopening, the stored set and the export matched on the new Concrete mixer row, the missing Site fencing row, and the unchanged header and totals.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Extend AIPassed1 of 1 test case passedLet a user add and remove invoice items through Human Review.View Result →

After approval, the traffic cones were present, the site fencing was absent and the telehandler and generator values were unchanged.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

NanonetsPassed1 of 1 test case passedLet a user add one line item and remove another while preserving the other two rows.View Result →

Both corrections persisted after a full page reload and appeared in the JSON export.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (5): Datalab, Landing AI, Reducto, Retab, Unstract.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Review and Correction

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

Repair a record in Structured Document Extraction | AI Demos