Tool in benchmark · Version 1

Nanonets in Structured Document Extraction

Scenario-level performance from current published Results.

11Passed
1Failed
8No published Result yet

12 of 20 scenarios published

CapabilityScenariosResult
Field Extraction3 pass · 0 fail · 3 of 5 published
Nested & Repeated Fields3 pass · 0 fail · 3 of 4 published
OCR1 pass · 0 fail · 1 of 2 published
Querying1 pass · 1 fail · 2 of 3 published
Review and Correction2 pass · 0 fail · 2 of 3 published
Source Grounding1 pass · 0 fail · 1 of 2 published
Training0 of 1 published
PassedFailedPublished, different test setNo published Result yet

Where Nanonets failed

1 of 12 graded scenarios. Open a Result to see the conversation and the evidence.

Querying

The answer was never in the schema

Nanonets presented a document answer as extracted data without distinguishing its source. The first request correctly returned the receipt number, subtotal, tax and total due. The follow-up used the same extraction tool, placing the technician in a separate, similarly styled extraction block; the numbered source chip appeared only in the answer text below, which did not say the technician was outside the requested fields.

View Result →

How Nanonets performed

Open a capability to explore its scenarios. Each row reports the test set in its published Result.

Field Extraction3 pass2 no Result yet
ScenarioPublished outcomesTest coverageResult
The value is directly available
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Nanonets passed this single-field extraction task. The receipt number appeared once on the document and returned as RHS-70234 in both the preview table and exported JSON; the other returned fields also matched the document.

1/1 assessed1/1 gradablePublished test setView Result →
The value must be derived
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Nanonets derived the requested value from the two present inputs and returned 81 for total_fuel_cost. The two operand rows carried source-link icons in the screenshot and the total row did not, matching the JSON output and showing the value as a computed field.

1/1 assessed1/1 gradablePublished test setView Result →
The value needs a supplied definition
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Nanonets chose the consignment reference, not the invoice number, for this field. The result row and exported JSON both show CFS-CN-40821, matching the supplied definition.

1/1 assessed1/1 gradablePublished test setView Result →
The field is absentNo published result
The same field across layoutsNo published result
Nested & Repeated Fields3 pass1 no Result yet
ScenarioPublished outcomesTest coverageResult
The set contains a non-record
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Nanonets passed this lookalike-row scenario. It returned five genuine line items and left out the lookalike Total row; the input table shows that row in the same layout, and the output has no 'Total' entry or 1,982 amount.

1/1 assessed1/1 gradablePublished test setView Result →
The set continues across a page break
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Nanonets kept one array across the page break. The output has ten objects in one top-level array, with no second array and no repeated member at the boundary. The invoice shows items 1-6 on page 1 and 7-10 on page 2, and the result screen shows the same single table numbered 1 to 10.

1/1 assessed1/1 gradablePublished test setView Result →
The set is legitimately empty
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Nanonets returned the empty set the document called for. The input says no damage or exceptions were reported, and the output is an empty array; the capture shows the same field as List [0] with no rows.

1/1 assessed1/1 gradablePublished test setView Result →
A repeated set of recordsNo published result
OCR1 pass1 no Result yet
ScenarioPublished outcomesTest coverageResult
A document mixing digital and scanned pages
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Nanonets handled the mixed digital-and-scanned document in this test. It returned the five requested values, including the two that appear only in the scanned page image, and kept the digital-page values unchanged. The JSON output itself does not show per-value page provenance.

1/1 assessed1/1 gradablePublished test setView Result →
A cleanly scanned documentNo published result
Querying1 pass1 fail1 no Result yet
ScenarioPublished outcomesTest coverageResult
Filter records across documents
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Nanonets returned the matching documents for the tested cross-document filter. It produced the two receipts above $150 and left out the one below that threshold. The screenshot also shows a Data Extraction step over three files before the final JSON answer.

1/1 assessed1/1 gradablePublished test setView Result →
The answer was never in the schema
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

Nanonets presented a document answer as extracted data without distinguishing its source. The first request correctly returned the receipt number, subtotal, tax and total due. The follow-up used the same extraction tool, placing the technician in a separate, similarly styled extraction block; the numbered source chip appeared only in the answer text below, which did not say the technician was outside the requested fields.

1/1 assessed1/1 gradablePublished test setView Result →
Aggregate across documentsNo published result
Review and Correction2 pass1 no Result yet
ScenarioPublished outcomesTest coverageResult
Corrected data goes downstream
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Nanonets passes this scenario. In the tested case, the corrected receipt number appears in the product’s delivery path: the result grid shows RHS-99999, the Download menu is open on that grid, and the exported JSON carries RHS-99999 with the other receipt values.

1/1 assessed1/1 gradablePublished test setView Result →
Repair a record
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Nanonets let a user add one line item and remove another while preserving the other two rows. Both corrections persisted after a full page reload and appeared in the JSON export.

1/1 assessed1/1 gradablePublished test setView Result →
Correct a field valueNo published result
Source Grounding1 pass1 no Result yet
ScenarioPublished outcomesTest coverageResult
Trace a value to its location
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Nanonets traced the returned subtotal to page 2, where GBP 6,103.00 is highlighted in the totals block. The exported page number was 2, and the preview highlight sat on the returned value rather than the neighboring totals.

1/1 assessed1/1 gradablePublished test setView Result →
The field is absentNo published result
Training1 no Result yet
ScenarioPublished outcomesTest coverageResult
A field the tool can only get right from examplesNo published result

Reading these Results

Published evidence and test coverage answer different questions.

Publication availability

Which scenarios have a Result?

A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.

Test coverage

What does each Result cover?

Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.

Scenario scope

Out of 19 scenarios

The summary counts each of the 19 scenarios once. A scenario fails when any of its published test cases failed. “No published Result yet” means nothing is public for that scenario; it does not say whether Nanonets was tested on it.

Nanonets in Structured Document Extraction | AI Demos