Scenario in benchmark · Structured Document Extraction · Version 1

Filter records across documents

Filter a subset of the tool's extracted records across multiple documents, using one extracted record per source document as the operating grain.

Capability: Querying1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the tool filters records from its own extracted set across multiple documents.
  2. Whether the result keeps one extracted record per source document as the operating grain.
  3. Whether only matching records are returned and non-matching records are excluded.
  4. Whether the operation happens inside the tool over extracted records rather than by exporting data elsewhere.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 2
Extend AIFailed1 of 1 test case failedOffered no way to filter records across the three extracted receipts.View Result →

It returned separate document extractions or one merged extraction, not a filtered set.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Landing AIFailed1 of 1 test case failedFailed to filter records across extracted documents.View Result →

Chat refused the request, and the interface only showed one document at a time with no filter or query control; the receipts themselves were extracted correctly, including two above 150 and one below.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 2
FutureSmart Document IntelligencePassed1 of 1 test case passedPasses this cross-document filter task.View Result →

It returned Meridian Retail Group invoice MRG-INV-20481 at £6,591.24. The run covered 17 documents, and a later 20-document totals table still shows that Blackwood invoice was the only record above £2,500.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

NanonetsPassed1 of 1 test case passedReturned the matching documents for the tested cross-document filter.View Result →

It produced the two receipts above $150 and left out the one below that threshold. The screenshot also shows a Data Extraction step over three files before the final JSON answer.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (6): Datalab, Docsumo, LlamaParse, Reducto, Retab, Unstract.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Querying

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

Filter records across documents in Structured Document Extraction | AI Demos