Mis-merges two independent side-by-side lists into one table, forcing unrelated cost-of-sales and SG&A bullets into fake row pairs where no row correspondence exists in the source.
What was measured
Table Preservation
Preserves complex table structures, including rows, columns, multi-row headers, and merged-cell relationships in markdown.
decisive for this rankingtransformation
Accurate Markdown conversion of complex PDFs depends on keeping table structure intact, not flattening it into plain text. (3 of 3 judges)
What was given, what came back
Test input: Hybrid Earnings Report · pdf · group: hybrid-earnings-report
Input — what we sent
Hybrid Earnings Report
A long, real-world hybrid annual report used to benchmark end-to-end PDF-to-markdown conversion. It combines native text, financial tables, charts/graphics, and scanned signature/stamp regions, stressing preservation of document structure, reading order, and embedded visual content across a multi-section report.
Why this input is hard
- · Native digital text extraction
- · Complex financial table preservation
- · Chart and graphic retention
- · Image-based signatures and stamps
- · Reading order across a long multi-section report
- · Markdown quality and consistency
Output — unretouched
Also checked on this input — same tool, 10 other criteria
Advanced Features◐ MixedThe parse pipeline’s confidence field catches all 10 footer-number corruptions and also produces one false alarm on a correct securities table, so the signal is real but not perfectly calibrated.Advanced Features✓ WorkedThe separate schema-driven extract.run endpoint cleanly recovers both a financial table and a donut chart: all requested table rows and all five donut percentages come back exactly.Advanced Features✗ FailedOn the bar-chart schema, extract.run gets only 3 of 5 Sales values right: 2011 and 2012 are silently pulled from a different table ($69.9B and $73.3B), with no confidence or warning signal.Complex Document Handling◐ MixedProcesses the full 84-page mixed-content report in 25.5 seconds with no truncation and keeps the hardest nested table’s 27 values exact, but 10 footer numbers are corrupted in the back two-thirds of the file.Markdown Quality⚠ StruggledProduces usable but not clean markdown: the output contains 12 <signature> tags, 2 <empty> placeholders, and literal HTML tags such as <b>, <i>, and <u> instead of pure CommonMark.Reading Order & Structure⚠ StruggledDrops heading structure on most of the filing: the FORM 10-K opening area receives no # or ## markup, and the report says 80 of 84 pages stay as plain paragraphs rather than navigable section headings.Reading Order & Structure✓ WorkedLinearizes a two-column page correctly, reading the full left column, including all four bullets, before the right-column continuation appears, with no interleaving or sequence inversion.Text & OCR Completeness✓ WorkedConverts the full 84-page native-digital annual report with no skipped pages; the output contains all 84 page-marker pairs, and checked high-risk numbers survive exactly, including the $51,550,988,273 market-value figure and 599,982,121 shares outstanding.Visual Content Retention✓ WorkedRetains visual content as real image outputs when requested: the portrait, donut chart, and bar chart are all returned as preserved figure/image crops rather than being dropped.Visual Content Retention✗ FailedDoes not retain handwritten signature graphics as images: a signature crop returns only the <signature> placeholder plus the printed name/title, omitting the cursive mark itself.
Provenance
- Observation
- 0521f27e-23a0-457d-a653-8d443d84ee9b
- Evidence run
- 6e3160de-fe46-4b45-b071-72560b5c5d0e
- Study
- Convert a Complex PDF into Clean Markdown with an API
- Research task
- 86b9h7t37
- Tested at
- not recorded
- Source
- first-party
- Evidence state
- verified
- Proof shown
- input + output shown
- Cost / latency
- not captured
- Repeat run
- not captured
- Tester
- not captured
The last three rows are honest blanks, not placeholders — our capture has no field for them yet.
Query this
get_evidence({
tool: "reducto",
scenario: "hybrid-earnings-report"
})MCP · mcp.aidemos.com/api/mcp
Free with attribution.
Same input, same check — 8 other tools
measured on Table Preservation
Adobe API✓ WorkedPreserves a 5-year financial summary table with the 2015–2011 columns and the full set of financial-result rows, keeping the row and column relationships intact.Extend AI✓ WorkedPreserves a 5-year financial summary table with row labels and year columns aligned, keeping the annual financial grid intact in markdown.Landing AI✓ WorkedReconstructs a five-year financial results table with the 2015–2011 columns and aligned row/value relationships intact in markdown-style rows.LlamaParse✓ WorkedPreserves aligned rows, columns, and value associations in a standard multi-year financial summary table.Mistral AI✓ WorkedThe financial table is reconstructed into a usable structured layout without losing the relationships between headers, rows, and corresponding values.Nutrient.io✗ FailedBreaks a 5-year financial summary table, misaligning rows, columns, and values instead of reconstructing the original structure cleanly.Tensorlake✓ WorkedKeeps a 5-year financial summary table structured, retaining the row/column relationships across sales, expenses, EBIT, and per-share rows instead of flattening the table.Upstage AI◐ MixedReconstructs the 2011–2015 financial summary table with rows and columns intact, but misses a small number of currency symbols in the extracted values.
This evidence is published in
Real inputs and real outputs, no retouching · every cell queryable via API & MCP · aidemos.com