It does not preserve the declared schema order, so the JSON is not directly consumable by order-sensitive pipelines without extra reshaping.
What was measured
Structural Clean Output
Is the JSON directly consumable by a downstream AI pipeline or system without requiring a structural transformation layer?
context, not decisivetransformation
Being directly consumable by a downstream pipeline is valuable, but it is more about integration convenience than whether the tool actually extracts the data correctly. (3 of 3 judges)
What was given, what came back
Test input: Invoice PDF · pdf · group: business-document-extraction
Input — what we sent
Invoice PDF
A two-page broadcast advertising invoice PDF with nested header metadata, billing and remit addresses, and eight line items spanning a page break. It was used to stress hierarchical line-item extraction, amount precision, time/day parsing, code extraction, and summary validation.
Why this input is hard
- · Nested line-item hierarchy extraction
- · Multi-page continuity across a page break
- · Precision on large dollar amounts and totals
- · Parsing complex time slots and day patterns
- · Extraction of Ad IDs and reconciliation codes
- · Mapping structured metadata sections correctly
- · Financial summary validation
- · Handling political advertising compliance text
Output — unretouched
Also checked on this input — same tool, 4 other criteria
Extraction Accuracy✗ FailedIt over-reads the payment-term field, outputting `Payment Terms 30 Days` instead of the requested `30 Days`.Schema Adherence✓ WorkedThe tool reconstructs the invoice hierarchy into nested JSON with invoice_metadata, advertiser, station, summary, and line_items objects rather than returning raw OCR.Semantic Field Enrichment✓ WorkedThe tool correctly derives higher-level line-item fields such as `day_of_week: Tu` and `days_pattern: -T----` alongside ad ID, reference number, and flight-period dates.Table & Record Completeness✓ WorkedIt extracts all 8 advertising line items as separate records, preserving item boundaries across the page break without merging adjacent rows.
Provenance
- Observation
- cbdef271-b824-418f-afc9-3031c0b079ab
- Evidence run
- a061b9e7-a9c5-443d-a171-b296aaf51b8c
- Study
- Extract and query structured data from documents using natural language
- Research task
- 86b9y25e5
- Tested at
- not recorded
- Source
- first-party
- Evidence state
- verified
- Proof shown
- input + output shown
- Cost / latency
- not captured
- Repeat run
- not captured
- Tester
- not captured
The last three rows are honest blanks, not placeholders — our capture has no field for them yet.
Query this
get_evidence({
tool: "extend-ai",
scenario: "business-document-extraction"
})MCP · mcp.aidemos.com/api/mcp
Free with attribution.
Same input, same check — 3 other tools
measured on Structural Clean Output
Datalab◐ MixedThe invoice JSON is structurally usable but not order-stable: the top-level keys are emitted in a fully shuffled order instead of the requested schema order.Landing AI✓ WorkedReturns directly consumable JSON for the invoice workflow, with the demo reaching extracted results and the report stating the JSON is immediately usable for integration.Retab⚠ StruggledPreserves valid JSON but does not keep the schema's original key order, so downstream consumers that expect schema-consistent ordering need an extra formatting step.
This evidence is published in
Real inputs and real outputs, no retouching · every cell queryable via API & MCP · aidemos.com