Graded 28 October 2026
Does LlamaParse extract cleanly scanned documents accurately?
LlamaParse passed this OCR baseline. It returned the same invoice values for the digital file and the image-only twin, including the page 2 totals, and the twin had no text layer. The two runs differed only in job details such as timestamps and confidence flags.
1 of 1 test case passed
Every test case in this scenario has a result.
Pass rate100%1 of 1 gradable
Coverage1 of 1test case with a result
1PassDid everything it was expected to do.
0FailNo test case failed.
0Not gradableEvery result here could be graded.
0UntestedEvery test case in this scenario has a result.
The pass rate is a summary. The evidence is the test case below: what we sent, what we checked, what the tool returned, and the proof.
The test case
Each test case is judged on its own: Pass, Fail, Not gradable, or Untested. The scenario result above is based on this test case.
Image-only twin, same schemaPassEvidence
What we sent
Input 1Document (PDF)
Input 2Document (PDF)
What the tool returned
JSONline 1, lines 9–38, line 503 of the file · copied lines 9–38
{ … "extract_result": { "invoice_number": "TPH-3301", "bill_to": "Oakbridge Construction Ltd\nSite Office, Oakbridge Depot Extension\nDarrowby DL4 7HN", "site": "Oakbridge Depot Extension", "hire_period": "03 Mar - 10 Mar 2026", "currency": "£", "subtotal": "£1,462.00", "tax": "£292.40", "total_due": "£1,754.40", "line_items": [ { "description": "Telehandler, 4-tonne, weekly hire", "quantity": "1", "unit_price": "£850.00", "amount": "£850.00" }, { "description": "Generator, 20kVA, weekly hire", "quantity": "2", "unit_price": "£210.00", "amount": "£420.00" }, { "description": "Site fencing, per 10m panel, weekly", "quantity": "8", "unit_price": "£24.00", "amount": "£192.00" } ] }, … }
Digital run output: extracted invoice fields and timestamps
Copied from Output file 1 (JSON), lines 9–38
Open the JSON file ↗Output file 2 (JSON)
The tool’s response is printed above, under What the tool returned. Show this file
JSON
{ "file_input": "f1c0ea86-0ac4-46cd-9f73-d5b9f4c08590", "id": "ext-cimhtanf5elr38mh083pxe3s0t9c", "project_id": "205fc038-050f-4b9d-9896-b404e84e1434", "configuration_id": null, "configuration": null, "status": "COMPLETED", "error_message": null, "extract_result": { "invoice_number": "TPH-3301", "bill_to": "Oakbridge Construction Ltd\nSite Office, Oakbridge Depot Extension\nDarrowby DL4 7HN", "site": "Oakbridge Depot Extension", "hire_period": "03 Mar - 10 Mar 2026", "currency": "£", "subtotal": "£1,462.00", "tax": "£292.40", "total_due": "£1,754.40", "line_items": [ { "description": "Telehandler, 4-tonne, weekly hire", "quantity": "1", "unit_price": "£850.00", "amount": "£850.00" }, { "description": "Generator, 20kVA, weekly hire", "quantity": "2", "unit_price": "£210.00", "amount": "£420.00" }, { "description": "Site fencing, per 10m panel, weekly", "quantity": "8", "unit_price": "£24.00", "amount": "£192.00" } ] }, "extract_metadata": { "field_metadata": { "document_metadata": { "invoice_number": { "citation": [ { "page": 1, "matching_text": "TPH-3301", "bounding_boxes": [ { "x": 488.1944425821305, "y": 153.7197560119629, "w": 39.33231800079346, "h": 8.280487976074218 } ], "page_dimensions": { "width": 594.96, "height": 841.92 } } ], "confidence": 0.9335547308743807, "extraction_confidence": 0.9335547308743807 }, "bill_to": { "citation": [ { "page": 1, "matching_text": "bill_to", "bounding_boxes": [ { "x": 52.891180893409995, "y": 164.4375, "w": 143.14827067669174, "h": 34.46609999999999 } ], "page_dimensions": { "width": 594.96, "height": 841.92 } } ], "confidence": 0.8843069763368065, "extraction_confidence": 0.8843069763368065 }, "site": { "citation": [ { "page": 1, "matching_text": "Oakbridge Depot Extension", "bounding_boxes": [ { "x": 91.90529554367066, "y": 176.87255902290343, "w": 103.53851634979249, "h": 11.437684965133666 } ], "page_dimensions": { "width": 594.96, "height": 841.92 } } ], "confidence": 0.944939244463218, "extraction_confidence": 0.944939244463218 }, "hire_period": { "citation": [ { "page": 1, "matching_text": "03 Mar - 10 Mar 2026", "bounding_boxes": [ { "x": 466.09063508033756, "y": 231.59735701560973, "w": 80.90848259925843, "h": 8.80667492866516 } ], "page_dimensions": { "width": 594.96, "height": 841.92 } } ], "confidence": 0.9006638092635528, "extraction_confidence": 0.9006638092635528 }, "currency": { "citation": [ { "page": 1, "matching_text": "currency", "bounding_boxes": [ { "x": 427.3396904024768, "y": 283.88489999999996, "w": 5.525944272445819, "h": 6.840600000000041 } ], "page_dimensions": { "width": 594.96, "height": 841.92 } } ], "confidence": 0.7244274285417167, "extraction_confidence": 0.7244274285417167 }, "subtotal": { "citation": [ { "page": 2, "matching_text": "£1,462.00", "bounding_boxes": [ { "x": 507.2984164810181, "y": 90.20741510868072, "w": 39.963809890151026, "h": 11.01672646433115 } ], "page_dimensions": { "width": 594.96, "height": 841.92 } } ], "confidence": 0.9126306960733258, "extraction_confidence": 0.9126306960733258 }, "tax": { "citation": [ { "page": 2, "matching_text": "£292.40", "bounding_boxes": [ { "x": 514.876833357811, "y": 111.46589741706848, "w": 32.122284321784974, "h": 10.964114198684692 } ], "page_dimensions": { "width": 594.96, "height": 841.92 } } ], "confidence": 0.9008971713028409, "extraction_confidence": 0.9008971713028409 }, "total_due": { "citation": [ { "page": 2, "matching_text": "£1,754.40", "bounding_boxes": [ { "x": 484.44, "y": 143.92, "w": 51.839999999999975, "h": 12.629999999999995 } ], "page_dimensions": { "width": 594.96, "height": 841.92 } } ], "confidence": 0.8897987451017179, "extraction_confidence": 0.8897987451017179 }, "line_items": [ { "description": { "citation": [ { "page": 1, "matching_text": "Telehandler, 4-tonne, weekly hire", "bounding_boxes": [ { "x": 58.74963305175305, "y": 279.4815489864349, "w": 121.69518370628357, "h": 11.174604034423828 } ], "page_dimensions": { "width": 594.96, "height": 841.92 } } ], "confidence": 0.9655682117325877, "extraction_confidence": 0.9655682117325877 }, "quantity": { "citation": [ { "page": 1, "matching_text": "1", "bounding_boxes": [ { "x": 358.9925823926926, "y": 282.37563995361324, "w": 4.597839345932007, "h": 8.280513067245483 } ], "page_dimensions": { "width": 594.96, "height": 841.92 } } ], "confidence": 0.9480293474137156, "extraction_confidence": 0.9480293474137156 }, "unit_price": { "citation": [ { "page": 1, "matching_text": "£850.00", "bounding_boxes": [ { "x": 427.145885424614, "y": 283.1649329280853, "w": 30.12245435714722, "h": 8.543619089126587 } ], "page_dimensions": { "width": 594.96, "height": 841.92 } } ], "confidence": 0.9553235831089298, "extraction_confidence": 0.9553235831089298 }, "amount": { "citation": [ { "page": 1, "matching_text": "£850.00", "bounding_boxes": [ { "x": 508.7193585634232, "y": 283.95425099372864, "w": 30.385549750328067, "h": 8.543619089126587 } ], "page_dimensions": { "width": 594.96, "height": 841.92 } } ], "confidence": 0.9583478109436722, "extraction_confidence": 0.9583478109436722 } }, { "description": { "citation": [ { "page": 1, "matching_text": "Generator, 20kVA, weekly hire", "bounding_boxes": [ { "x": 58.64437627673149, "y": 303.5815186214447, "w": 110.48541030764581, "h": 12.595334368050098 } ], "page_dimensions": { "width": 594.96, "height": 841.92 } } ], "confidence": 0.9638471239436878, "extraction_confidence": 0.9638471239436878 }, "quantity": { "citation": [ { "page": 1, "matching_text": "2", "bounding_boxes": [ { "x": 358.2031543636322, "y": 307.6332409572601, "w": 5.913529086112977, "h": 8.543619089126587 } ], "page_dimensions": { "width": 594.96, "height": 841.92 } } ], "confidence": 0.9444230769297934, "extraction_confidence": 0.9444230769297934 }, "unit_price": { "citation": [ { "page": 1, "matching_text": "£210.00", "bounding_boxes": [ { "x": 426.8827545690537, "y": 308.4225339317322, "w": 30.122418894767762, "h": 8.543619089126587 } ], "page_dimensions": { "width": 594.96, "height": 841.92 } } ], "confidence": 0.9662071433827945, "extraction_confidence": 0.9662071433827945 }, "amount": { "citation": [ { "page": 1, "matching_text": "£420.00", "bounding_boxes": [ { "x": 508.2456875610352, "y": 308.47515011787414, "w": 30.596089897155764, "h": 9.28032096862793 } ], "page_dimensions": { "width": 594.96, "height": 841.92 } } ], "confidence": 0.9690494579067602, "extraction_confidence": 0.9690494579067602 } }, { "description": { "citation": [ { "page": 1, "matching_text": "Site fencing, per 10m panel, weekly", "bounding_boxes": [ { "x": 58.11809683442116, "y": 329.1022256469726, "w": 130.48406457781792, "h": 12.595334368050098 } ], "page_dimensions": { "width": 594.96, "height": 841.92 } } ], "confidence": 0.9700691456200681, "extraction_confidence": 0.9700691456200681 }, "quantity": { "citation": [ { "page": 1, "matching_text": "8", "bounding_boxes": [ { "x": 357.94002350807193, "y": 333.15394798278805, "w": 6.176659941673279, "h": 8.543619089126587 } ], "page_dimensions": { "width": 594.96, "height": 841.92 } } ], "confidence": 0.9493916734928425, "extraction_confidence": 0.9493916734928425 }, "unit_price": { "citation": [ { "page": 1, "matching_text": "£24.00", "bounding_boxes": [ { "x": 430.829859251976, "y": 333.9432409572601, "w": 25.912183356285098, "h": 8.543619089126587 } ], "page_dimensions": { "width": 594.96, "height": 841.92 } } ], "confidence": 0.966369988144164, "extraction_confidence": 0.966369988144164 }, "amount": { "citation": [ { "page": 1, "matching_text": "£192.00", "bounding_boxes": [ { "x": 508.1930968523026, "y": 334.7325339317322, "w": 30.385549750328067, "h": 8.543619089126587 } ], "page_dimensions": { "width": 594.96, "height": 841.92 } } ], "confidence": 0.9701031466390181, "extraction_confidence": 0.9701031466390181 } } ] }, "page_metadata": null, "row_metadata": null }, "parse_job_id": "pjb-m16vuqep6b2bgefdoioy78yxvbjs", "parse_tier": "agentic" }, "metadata": null, "grounded_items": null, "usage": null, "created_at": "2026-10-28T06:00:53.796482Z", "updated_at": "2026-10-28T06:01:24.570225Z", "document_input_value": "f1c0ea86-0ac4-46cd-9f73-d5b9f4c08590" }
Expected vs. Found
| ✓ | Expected: The two runs should return the same values and report the difference.Found: Both runs matched on all extracted fields and line items, and the page 2 totals were read from the twin. |
Supporting proof
Proof 1Screenshot
Proof 2Screenshot
Why this result
The digital file and the image-only twin returned matching invoice values under the same schema, and the twin had no text layer. That shows the OCR path was tested, not hidden text.
Also observed in this test case
Timing difference. The digital run finished in 25.6 seconds, and the twin run in 30.8 seconds.
Confidence flags. At the 90% threshold, the digital run flagged 5 of 20 fields and the twin flagged 3; every flagged value was correct.
Tested by Rugved Nichite · evidence dated 28 October 2026
Configuration and setup
How this tool was set up for the run and what the test needed in place. Each row is a fact from the run's records; a fact the records do not hold is left out, not guessed.
Software that produced the output
LlamaParse
Build or version
LlamaCloud web app (build v9bc3d1a1); Extract, Agentic tier
Plan or tier
Free plan (20,000 credits)
Surface
Web app
Set up before the run
A matched digital invoice and scanned twin were required, with the twin carrying no text layer. One unchanged schema was run on the digital file and then on the twin.
Extraction tier
Agentic
Extraction schema
Run (LlamaCloud / LlamaParse web app, Extract page, Agentic extraction tier; one JSON schema used unchanged for both runs, 9 fields: invoice_number, bill_to, site, hire_period, currency, subtotal, tax, total_due (all strings) and line_items (an array of description, quantity, unit_price, amount))
Confidence threshold
90%
Tested
28 October 2026 · Rugved Nichite
How this scenario is graded
How we decide Pass, Fail and Not gradable. The same rules apply to every tool tested on this scenario.
How results are decided
Each test case gets one result per tool: Pass, Fail or Not gradable. A test case we haven't run yet shows Untested. There are no partial results.
The rules
- Pass: the tool did everything the test expected, and nothing it said contradicts the correct answer.
- Fail: at least one expected behaviour clearly didn't happen; the test case's result says which one and quotes the tool.
- Not gradable: our evidence couldn't settle the outcome (for example a record we needed is missing). It is never counted as a fail, and the test case's result says what's missing.
Where this sits in the benchmark
This page is one cell of a larger study: one tool, one scenario. Only this benchmark's frame appears here.
| Level | Name | Scope |
|---|---|---|
| Benchmark | Structured Document Extraction → | 19 scenarios · 10 tools |
| Capability | OCR → | |
| Scenario | A cleanly scanned document → | |
| Tool | LlamaParse → |
Global scenario definition → · Global capability definition → · LlamaParse product page →
History of this result
What has happened to this result since it was first published. Runs and grades are never overwritten: a retest or a re-grade publishes a new result and keeps the earlier one readable.
30 October 2026First publishedStructured Document Extraction v1
Act on this result
Nothing filed here edits the run or the grade. A challenge opens a review, and a review can produce a new run or a re-grade — which becomes the current result and leaves this one in the history.
This matches what I see
You run the same kind of test against your own setup and get the same behaviour.
Agree →This does not match
Yours behaves differently. Tell us what you got, with a screenshot if you have one.
Disagree →Point out an issue
Something here is wrong — a reference value, a transcription, a grade.
Report an issue →We have fixed this
Tell us what changed and we schedule a rerun of the failing test case. The old result stays as history.
Vendor notice →Filed against this evidenceNothing yet. Challenges, counter-evidence and fix notices appear here with their outcome, and stay on the page after they are resolved.
Cite this result
aidemos.com/benchmarks/structured-document-extraction/results/llamaparse/a-cleanly-scanned-document · 1 pass · 0 fail · coverage 1/1 · graded 2026-10-28
The same record is available as structured data through the AI Demos MCP server, with the counts, the coverage and every per-test-case reason carried as fields.
Verify the proof files
These files support this result. Open a file to inspect the original evidence.
Input 1 · Document (PDF) ↗application/pdf · 78 KB
Input 2 · Document (PDF) ↗application/pdf · 232 KB
Output file 1 (JSON) ↗text/plain; charset=utf-8 · 15 KB
Output file 2 (JSON) ↗text/plain; charset=utf-8 · 15 KB
Proof 1 · Screenshot ↗image/png · 119 KB
Proof 2 · Screenshot ↗image/png · 345 KB
File fingerprints (SHA-256)
A fingerprint identifies the exact file used for this result.
Input 1 · Document (PDF)fdcf03ac5ee53b083da58166f17fd41e7c72de79a9f3a3f6b67daa1360ddbca9
Input 2 · Document (PDF)6f1cc2a5f5e5e57494671341557d7a53d318b0f62bb8f8536165e1dac1b7a805
Output file 1 (JSON)09e6f154e171bfc27542083442e760c00988a6491b2a3238c71c1394fef726a4
Output file 2 (JSON)dfa6cfb94d5d2c1188a7f7cfcba5af5181978fe2cf4c8392d66ecc86a8655ce3
Proof 1 · Screenshot3b265bff05f8fc482e7b2dd9df216978c64322064274b8b56ee6632e76a8066f
Proof 2 · Screenshotfbe8f2de85661e2187c13e505eb31d418ff3339d4f9245185cbfb99608b9c0b9