The invoice JSON is structurally usable but not order-stable: the top-level keys are emitted in a fully shuffled order instead of the requested schema order.
What was measured
Structural Clean Output
Is the JSON directly consumable by a downstream AI pipeline or system without requiring a structural transformation layer?
context, not decisivetransformation
Being directly consumable by a downstream pipeline is valuable, but it is more about integration convenience than whether the tool actually extracts the data correctly. (3 of 3 judges)
What was given, what came back
Test input: Invoice PDF · pdf · group: business-document-extraction
Input — what we sent
Research media invoice schema order.png
Invoice PDF
A two-page broadcast advertising invoice PDF with nested header metadata, billing and remit addresses, and eight line items spanning a page break. It was used to stress hierarchical line-item extraction, amount precision, time/day parsing, code extraction, and summary validation.
Why this input is hard
- · Nested line-item hierarchy extraction
- · Multi-page continuity across a page break
- · Precision on large dollar amounts and totals
- · Parsing complex time slots and day patterns
- · Extraction of Ad IDs and reconciliation codes
- · Mapping structured metadata sections correctly
- · Financial summary validation
- · Handling political advertising compliance text
Also checked on this input — same tool, 4 other criteria
Extraction Accuracy✓ WorkedRecovers invoice metadata and financial totals accurately: invoice number 4064621-1, invoice date 10/28/12, gross total 29750, agency commission 4462.5, net amount due 25287.5, and 8 aired spots match the source.Schema Adherence✓ WorkedThe invoice workflow can emit nested schema-shaped JSON with separate invoice_metadata, advertiser, station, account_details, billing_address, remit_address, flight_dates, line_items, and summary objects rather than flat OCR text.Semantic Field Enrichment✓ WorkedCaptures derived invoice line-item fields correctly, including time_slot 9a-10a, day_of_week Su, air_time 9:38 AM, and ad_id NRCCWI071005 on line item 2.Table & Record Completeness✓ WorkedExtracts all eight invoice line items as separate records without merging, including rows that span the page break.
Provenance
- Observation
- d7313df2-02ba-4bef-aa11-d9889779422f
- Evidence run
- a061b9e7-a9c5-443d-a171-b296aaf51b8c
- Study
- Extract and query structured data from documents using natural language
- Research task
- 86b9y25e5
- Tested at
- not recorded
- Source
- first-party
- Evidence state
- verified
- Proof shown
- input + output shown
- Cost / latency
- not captured
- Repeat run
- not captured
- Tester
- not captured
The last three rows are honest blanks, not placeholders — our capture has no field for them yet.
Query this
get_evidence({
tool: "datalab",
scenario: "business-document-extraction"
})MCP · mcp.aidemos.com/api/mcp
Free with attribution.
Same input, same check — 3 other tools
measured on Structural Clean Output
Extend AI✓ WorkedThe invoice output includes per-section OCR confidence metadata alongside the extracted fields while remaining machine-readable JSON that can be consumed without structural transformation.Landing AI✓ WorkedReturns directly consumable JSON for the invoice workflow, with the demo reaching extracted results and the report stating the JSON is immediately usable for integration.Retab⚠ StruggledPreserves valid JSON but does not keep the schema's original key order, so downstream consumers that expect schema-consistent ordering need an extra formatting step.
This evidence is published in
Real inputs and real outputs, no retouching · every cell queryable via API & MCP · aidemos.com