AI Demos Research — the structured-intelligence platform. Every verdict on these pages opens to the execution behind it.
Graded 18 October 2026

Does Firecrawl read a multi-column page in the right order?

Firecrawl did not consistently preserve column reading order. In a three-page bulletin, it merged a left-column heading into a right-column heading on page 1; the remaining text followed the intended column order.

1 of 1 test case failed

Every test case in this scenario has a result.

Pass rate0%0 of 1 gradable
Coverage1 of 1test case with a result
0PassNo test case passed.
1FailMissed at least one thing it was expected to do; the reason and proof are with each test case below.
0Not gradableEvery result here could be graded.
0UntestedEvery test case in this scenario has a result.
The pass rate is a summary. The evidence is the test case below: what we sent, what we checked, what the tool returned, and the proof.

The test case

Each test case is judged on its own: Pass, Fail, Not gradable, or Untested. The scenario result above is based on this test case.

Two-column page — natural reading orderFailEvidence
What we sent
InputDocument (PDF)
Input PDF: three-page, two-column society bulletin
Open the PDF ↗
What the tool returned

The Vellamy Society Bulletin

Number 212 · March 2026

Spring lecture programme

Three lectures are arranged for the spring, all at the Assembly Rooms on Vardenne Quay at half past seven. On 11 March Dr Aileen Prosser will speak on the salt industry of the upper estuary; on 8 April Mr Colm Tarbert will describe the 1953 flood as it was recorded at Kellow; and on 13 May the Society's archivist, Miss Ruth Sandiman, will introduce the Harrowby field books, which the Society acquired last autumn.

Members may bring one guest without charge. Tea is served from seven o'clock, and the room is open from half past six for those who wish to look at the exhibits. Anyone needing a lift from the town should telephone the honorary secretary, who keeps a list of members willing to offer one.

The Harrowby field books

The eleven field books kept by Thomas Harrowby between 1871 and 1904 came to the Society in October, the gift of his great-granddaughter, and are now with the county record office for conservation. They are working notebooks rather than journals: tide times, wind, the state of the banks at Thrimby, the names of the men who worked each reach, and here and there a sketch of a channel that has since moved half a mile. The eighth book, which covers the winter of 1889, is the one most often asked for, because it records the breach at Kellow in some detail and gives the height of the water against the wall of the old lifeboat house.

The Society hopes to publish a transcript of the eighth book, and would welcome help with it. What is wanted is not palaeography - the hand is a clear one - but local knowledge: field names, the older names for the reaches, and the identification of men named only by a surname and an initial. Members who could give an evening a month to this should write to the archivist.

The summer excursion

The summer excursion will be to Marnhead on Saturday 20 June, to see the salt pans and what is left of the tramway on the north side of the harbour. Mr Tarbert has agreed to lead the walk, which is about three miles over rough ground and is not suitable for anyone who cannot manage a stile. The coach leaves the Assembly

Rooms at half past nine and is back by six. The cost, including the coach and the entry to the harbour office, is fourteen pounds, and members should book with the excursion secretary before 1 June.

New members and gifts

The Society welcomes seven new members since the autumn bulletin: three from Vardenne, two from Braithe, one from Kellow and one, to the committee's particular pleasure, from Winnipeg. The library has received the bound volumes of the Braithe Advertiser for 1928 to 1934 from the estate of the late Mr E. Lissamer, and an album of photographs of the 1947 regatta from Mrs Prosser of Thrimby. Both have been accessioned and may be consulted on the usual notice.

Notes and queries

Notes and queries The Harrowby field books

A member asks whether anyone can identify the building in the postcard reproduced in the autumn bulletin. The committee's view is that it is the old ferry office at Vardenne, pulled down about 1961, but the roof line does not match the photograph in the Society's own collection and the question is left open. Replies to the archivist, please, and not to the editor, who is away until April.

Subscriptions

Subscriptions for the year beginning 1 April are unchanged: eighteen pounds for a member, twenty-six pounds for two members at one address, and six pounds for members under twenty-five. Those who pay by standing order need do nothing. The Society remains grateful to the twenty-one members who add a donation to their subscription.

The old lifeboat house

Show the reply as returned (Markdown)
# The Vellamy Society Bulletin

Number 212  ·  March 2026

## Spring lecture programme

Three lectures  are  arranged  for  the  spring,  all  at  the
Assembly Rooms on Vardenne Quay at half past seven.
On 11  March  Dr  Aileen  Prosser will speak on the salt
industry of  the  upper  estuary;  on  8  April  Mr  Colm
Tarbert will describe the  1953  flood as it was recorded
at Kellow;  and on  13  May the  Society's archivist,  Miss
Ruth Sandiman,  will  introduce  the  Harrowby  field
books, which the Society acquired last autumn.

Members may  bring  one  guest  without  charge.  Tea  is
served from seven o'clock,  and the room is open from
half past six for those who wish to look at the exhibits.
Anyone needing a lift from the town should telephone
the honorary  secretary,  who  keeps  a  list  of  members
willing to offer one.

## The Harrowby field books

The eleven  field  books  kept  by  Thomas  Harrowby
between 1871 and 1904 came to the Society in October,
the gift  of  his  great-granddaughter,  and  are  now  with
the county  record  office  for  conservation.  They  are
working notebooks  rather  than  journals:  tide  times,
wind, the state of the banks at  Thrimby,  the names of
the men who worked each reach,  and here and there a
sketch of  a  channel  that  has  since  moved  half  a  mile.
The eighth  book,  which  covers  the  winter  of  1889,  is
the one  most  often  asked  for,  because  it  records  the
breach at Kellow in some detail and gives the height of
the water against the wall of the old lifeboat house.

The Society hopes to publish a transcript of the eighth
book, and would welcome help with it. What is wanted
is not palaeography - the hand is a clear one - but local
knowledge: field  names,  the  older  names  for  the
reaches, and the identification of men named only by a
surname and  an  initial.  Members  who  could  give  an
evening a month to this should write to the archivist.

## The summer excursion

The summer excursion will be to Marnhead on Saturday
20 June,  to  see  the  salt  pans  and  what  is  left  of  the
tramway on the north side of the harbour.  Mr  Tarbert
has agreed to lead the walk,  which is about three miles
over rough ground and is not suitable for anyone who
cannot manage a stile.  The coach leaves the  Assembly

Rooms at  half  past  nine  and  is  back  by  six.  The  cost,
including the coach and the entry to the harbour office,
is fourteen pounds, and members should book with the
excursion secretary before 1 June.

## New members and gifts

The Society  welcomes  seven  new  members  since  the
autumn bulletin:  three  from  Vardenne,  two  from
Braithe, one from  Kellow and one,  to the committee's
particular pleasure,  from  Winnipeg.  The  library  has
received the  bound  volumes  of  the  Braithe  Advertiser
for 1928  to  1934  from  the  estate  of  the  late  Mr  E.
Lissamer, and  an  album  of  photographs  of  the  1947
regatta from  Mrs  Prosser of  Thrimby.  Both have been
accessioned and may be consulted on the usual notice.

## Notes and queries

## Notes and queries The Harrowby field books

A member  asks  whether  anyone  can  identify  the
building in  the  postcard  reproduced  in  the  autumn
bulletin. The committee's view is that it is the old ferry
office at  Vardenne,  pulled  down  about  1961,  but  the
roof line does not match the photograph in the Society's
own collection and the question is left open. Replies to
the archivist, please, and not to the editor, who is away
until April.

## Subscriptions

Subscriptions for  the  year  beginning  1  April  are
unchanged: eighteen  pounds  for  a  member,  twenty-six
pounds for  two  members  at  one  address,  and  six
pounds for members under twenty-five. Those who pay
by standing order need do nothing. The Society remains
grateful to  the  twenty-one  members  who  add  a
donation to their subscription.

## The old lifeboat house

API response: returned Markdown and conversion metadata

Copied from Output file (JSON), data › markdown · an excerpt

Open the whole JSON file ↗
Output file (Markdown)
Returned Markdown: the three-page bulletin
Markdown308 lines
# The Vellamy Society Bulletin

Number 212  ·  March 2026

## Spring lecture programme

Three lectures  are  arranged  for  the  spring,  all  at  the
Assembly Rooms on Vardenne Quay at half past seven.
On 11  March  Dr  Aileen  Prosser will speak on the salt
industry of  the  upper  estuary;  on  8  April  Mr  Colm
Tarbert will describe the  1953  flood as it was recorded
at Kellow;  and on  13  May the  Society's archivist,  Miss
Ruth Sandiman,  will  introduce  the  Harrowby  field
books, which the Society acquired last autumn.

Members may  bring  one  guest  without  charge.  Tea  is
served from seven o'clock,  and the room is open from
half past six for those who wish to look at the exhibits.
Anyone needing a lift from the town should telephone
the honorary  secretary,  who  keeps  a  list  of  members
willing to offer one.

## The Harrowby field books

The eleven  field  books  kept  by  Thomas  Harrowby
between 1871 and 1904 came to the Society in October,
the gift  of  his  great-granddaughter,  and  are  now  with
the county  record  office  for  conservation.  They  are
working notebooks  rather  than  journals:  tide  times,
wind, the state of the banks at  Thrimby,  the names of
the men who worked each reach,  and here and there a
sketch of  a  channel  that  has  since  moved  half  a  mile.
The eighth  book,  which  covers  the  winter  of  1889,  is
the one  most  often  asked  for,  because  it  records  the
breach at Kellow in some detail and gives the height of
the water against the wall of the old lifeboat house.

The Society hopes to publish a transcript of the eighth
book, and would welcome help with it. What is wanted
is not palaeography - the hand is a clear one - but local
knowledge: field  names,  the  older  names  for  the
reaches, and the identification of men named only by a
surname and  an  initial.  Members  who  could  give  an
evening a month to this should write to the archivist.

## The summer excursion

The summer excursion will be to Marnhead on Saturday
20 June,  to  see  the  salt  pans  and  what  is  left  of  the
tramway on the north side of the harbour.  Mr  Tarbert
has agreed to lead the walk,  which is about three miles
over rough ground and is not suitable for anyone who
cannot manage a stile.  The coach leaves the  Assembly

Rooms at  half  past  nine  and  is  back  by  six.  The  cost,
including the coach and the entry to the harbour office,
is fourteen pounds, and members should book with the
excursion secretary before 1 June.

## New members and gifts

The Society  welcomes  seven  new  members  since  the
autumn bulletin:  three  from  Vardenne,  two  from
Braithe, one from  Kellow and one,  to the committee's
particular pleasure,  from  Winnipeg.  The  library  has
received the  bound  volumes  of  the  Braithe  Advertiser
for 1928  to  1934  from  the  estate  of  the  late  Mr  E.
Lissamer, and  an  album  of  photographs  of  the  1947
regatta from  Mrs  Prosser of  Thrimby.  Both have been
accessioned and may be consulted on the usual notice.

## Notes and queries

## Notes and queries The Harrowby field books

A member  asks  whether  anyone  can  identify  the
building in  the  postcard  reproduced  in  the  autumn
bulletin. The committee's view is that it is the old ferry
office at  Vardenne,  pulled  down  about  1961,  but  the
roof line does not match the photograph in the Society's
own collection and the question is left open. Replies to
the archivist, please, and not to the editor, who is away
until April.

## Subscriptions

Subscriptions for  the  year  beginning  1  April  are
unchanged: eighteen  pounds  for  a  member,  twenty-six
pounds for  two  members  at  one  address,  and  six
pounds for members under twenty-five. Those who pay
by standing order need do nothing. The Society remains
grateful to  the  twenty-one  members  who  add  a
donation to their subscription.

## The old lifeboat house

The Society's  letter  to  the  district  council  about  the
lifeboat house was answered in January. The building is
not listed  and  the  council  does  not  propose  to  seek
listing, but  the  officer  who  replied  confirmed  that  any
application to  demolish  it  would  come  before  the
conservation panel, on which the Society has a seat. The
nineteen members  who  wrote  in  support  of  the  letter
are thanked.

---

The committee's  view  remains  that  the  interest  of  the
place is the winch house and the slipway rather than the
shed itself,  which  was  re-roofed  in  the  1960s  and  has
lost its doors.  A  measured survey would settle what is
left of  the  original  fabric,  and  the  committee  is  asking
whether any member has the skill and the time to make
one.

## From the archive: the breach of 1889

The eighth  Harrowby field book gives the night of  12
December 1889  in  more  detail  than  any  other  source
the Society  holds.  Harrowby  was  at  Kellow  from  the
early evening  and  his  entries  are  timed:  the  wall  was
overtopping by  nine,  the  first  gap  was  made  at  about
half past  ten,  and  by  midnight  he  put  it  at  sixty  yards
and taking. He gives the height of the water against the
lifeboat house  wall  as  four  feet  nine  inches,  which  is
nine inches above the mark cut there after the flood of
1874.

What the book does not say is how the gap was closed,
and the  parish  records  are  no  better.  The  rate  book
shows a  payment  to  a  Marnhead  contractor  in  the
spring of  1890  and  nothing  else  at  all.  If  any  member
knows of  a  family  paper  that  mentions  the  work,  the
archivist would be glad to hear of it.

The Society intends to print the whole of the December
entries in the summer bulletin,  with a note on the tide
and the  wind  of  that  week.  The  transcript  is  finished
and is being checked against the original at the record
office.

## The photograph collection

Cataloguing has reached the end of the Vardenne boxes:
eleven hundred  prints  and  about  four  hundred
negatives, of  which  rather  more  than  half  are  now
identified at least to a street or a reach. The catalogue is
a card index and will stay one until the  Society can be
reasonably sure  that  whatever  replaces  it  could  be
maintained by whoever is doing the work in ten years'
time.

Copies can  be  supplied  to  members  for  a  charge
covering the  print  and  the  postage.  The  Society  does
not hold  the  copyright  in  every  image  and  will  say  so
where that  is  the  case;  where  the  photographer  is
unknown, prints are supplied on the usual undertaking.

## The estuary in 1926

A hundred years ago this spring the harbour board was
deciding whether to dredge the bar,  and the papers of

the argument  are  worth  reading.  The  board's  engineer
wanted a  channel  of  fourteen  feet  and  got  eleven;  the
fishermen wanted  the  bar  left  alone  and  were  not
consulted until  the  decision  had  been  taken;  and  the
compromise was  arrived  at  in  a  meeting  of  which  no
minutes were kept at all. The channel silted again within
four years.

The Society holds copies of the engineer's two reports
and of  the  petition  against  them,  which  carries  a
hundred and  ninety-one  signatures  and  is  the  best  list
we have  of  the  men  working  out  of  Vardenne  in  that
year. Members  are  welcome  to  consult  it,  and  a
transcript of the signatures is being prepared.

## Mr E. Lissamer

Members will  be  sorry  to  learn  of  the  death  in
November of  Mr  Edward  Lissamer,  who  joined  the
Society in 1974 and served on the committee for eleven
years, five  of  them  as  honorary  treasurer.  He  was  a
patient and exact man who kept the accounts in a hand
nobody could  misread,  and  he  had  a  memory  for  the
estuary's boats  that  the  Society  will  not  replace.  His
library has come to us under his will and will be listed in
the next bulletin.

## Visits and walks

Four walks are arranged for the summer, each of about
two hours and none of them strenuous. On 6 June, the
Vardenne quays with Mr Tarbert; on 4 July, the training
wall and  the  east  bank,  tide  permitting,  with  Miss
Sandiman; on 1 August, the Kellow marshes, led jointly
with the county wildlife trust;  and on  5  September,  the
town's chapels.  All  start  at  half  past  ten  from  the
Assembly Rooms  unless  the  programme  card  says
otherwise.

Members are  asked  to  book  for  the  marshes  walk,
which is limited to twenty and needs boots. The others
need no  booking  and  no  charge  is  made,  though  the
collection at the end goes to the room fund.

## Committee news

The annual general meeting will be held at the Assembly
Rooms on  Tuesday  19  May  at  seven  o'clock.  The
officers all  offer  themselves  for  re-election  except  the
honorary treasurer, who does not, and nominations for
that office  should  reach  the  secretary  in  writing  a
fortnight before the meeting. The accounts to 31 March
will be circulated with the summer bulletin.

The committee  has  agreed  to meet  the  increase  in the room hire  from  the  reserve  rather  than  from
subscriptions, and to review that at the annual general
meeting.

## Reviews

*A History  of  the  Marnhead  Salt  Trade*, by  J.  Cavenshaw
(Vardenne, 2025,  214  pp).  This is a careful book and a
slightly frustrating one. Cavenshaw has worked through
the harbour board's papers properly and his chapter on
the tramway is the best account we have. But he is not
much interested  in  the  pans  themselves,  and  a  reader
who wants  to  know  how  salt  was  actually  made  at
Marnhead will  have  to  go  back  to  Sandiman.
Recommended, with that reservation.

## Letters

Sir -  your correspondent's  note in the autumn  bulletin
on the  naming  of  Croyde  Gut  may  interest  members
who know the Ordnance sheets of 1897 and 1923. The
name appears  on  neither,  and  the  reach  is  called
Crowder's Gut on both. Whether Croyde is a correction
or a  corruption  I  cannot  say,  but  the  older  form  is  at
least the surveyed one. - R. Vale, Thrimby.

## Wanted: the Vardenne ferry

The Society  is  trying  to  establish  when  the  Vardenne
ferry stopped  running.  The  date  usually  given  is  1961,
which is the year the ferry office was pulled down,  but
the office  may  well  have  stood  empty  for  some  time
before that.  A  member  recalls  being  taken  across  as  a
child in the summer  of  1958  and is sure the boat was
still working  then;  another  remembers  it  ending  just
after the war. A timetable, a ticket, or a photograph with
a date on the back would settle it.

The same member has given the  Society a copy of the
1957 timetable for the Marnhead bus, which connected
with the  ferry  and  may  therefore  date  its  last  summer
more exactly  than  anything  else  we  hold.  It  is  on  the
notice board  at  the  Assembly  Rooms  for  anyone  who
wants to look at it.

## The chapels survey

The survey  of  the  town's  chapels,  begun  in  2019  and
interrupted twice,  is  now  complete  for  the  four
buildings still standing and for two of the three that are
not. What  remains  is  the  Bethel  on  Kellow  Lane,  of
which no  plan  and  only  one  photograph  have  been
found. The survey will be printed as an occasional paper
in the  autumn,  and  the  members  who  took  part  are

asked to check their entries before then.

The Society  is  grateful  to  the  trustees  of  two  of  the
chapels for  access  to  their  records,  and  to  the  county
archivist for the loan of a plan that had been catalogued
under the wrong parish for sixty years.

## Membership

Membership stood at two hundred and six at the end of
January, four more than a year ago and the first increase
since 2019.  Twenty-two  members  live  outside  the
county and six outside the country; the Society posts the
bulletin to  all  of  them  and  would  rather  do  that  than
lose them.  Anyone  who  would  prefer  it  by  e-mail
should tell the secretary,  who keeps a list and is happy
for it to grow.

## Notices

The Society's thanks go to the members who helped at
the December sale,  which raised a hundred and twelve
pounds for the room fund,  and to  Mrs  Tarbert for the
loan of the tables.

The room  fund  stood  at  three  hundred  and  eighty
pounds at  the  end  of  January.  The  committee  thanks
the members  who  have  given  to  it,  and  reminds
everyone that  the  fund  exists  to  keep  the  Assembly
Rooms available  to  the  Society  rather  than  to
accumulate.

Copy for  the  summer  bulletin  should  reach  the  editor
by 15 May. Short notes are as welcome as articles, and a
query that has been in the back of a member's mind for
years is often the most useful thing in an issue.
Open the Markdown file ↗
Expected vs. Found
✗Expected: Text follows the natural reading order without interleaving columns.Found: On page 1, a left-column heading appeared again inside the right-column Notes and queries heading. The rest of the three-page bulletin followed left-column-then-right-column order.
Supporting proof
Proof 1Screenshot
Source and output comparison: page 1
Open original ↗
Proof 2Screenshot
Source and output comparison: page 2
Open original ↗
Proof 3Screenshot
Source and output comparison: page 3
Open original ↗
Proof 4Screenshot
Run card: conversion conditions, timing and usage
Open original ↗
Why this result

The conversion had to read each column in turn. Firecrawl returned a full conversion, but a heading crossed between columns in its Markdown. That is a reading-order failure, rather than a failed request.

Also observed in this test case

Masthead text omitted. The page 1 masthead includes “Founded 1898”, but that line is absent from the returned Markdown.

Paragraph spacing retained. The bulletin's returned Markdown retained runs of double spaces between words from the source's justified text.

API usage reported. The bulletin's API response reported a successful status, three pages and three credits used.

Tested by Chandresh Bisht · evidence dated 10 September 2026

Configuration and setup

How this tool was set up for the run and what the test needed in place. Each row is a fact from the run's records; a fact the records do not hold is left out, not guessed.

Software that produced the output
Firecrawl
Build or version
not exposed (Firecrawl API v2, /v2/parse)
Plan or tier
free tier
Surface
API
Set up before the run
A short PDF with two clear text columns was required. The complete PDF was to be converted to Markdown in one pass using the default pipeline, without tool-specific flag tuning or cropping.
Pipeline
Default pipeline, single pass; no options set (the product's defaults) · Markdown format only
Tested
10 September 2026 · Chandresh Bisht

How this scenario is graded

How we decide Pass, Fail and Not gradable. The same rules apply to every tool tested on this scenario.

How results are decided

Each test case gets one result per tool: Pass, Fail or Not gradable. A test case we haven't run yet shows Untested. There are no partial results.

The rules
  • Pass: the tool did everything the test expected, and nothing it said contradicts the correct answer.
  • Fail: at least one expected behaviour clearly didn't happen; the test case's result says which one and quotes the tool.
  • Not gradable: our evidence couldn't settle the outcome (for example a record we needed is missing). It is never counted as a fail, and the test case's result says what's missing.

Where this sits in the benchmark

This page is one cell of a larger study: one tool, one scenario. Only this benchmark's frame appears here.

History of this result

What has happened to this result since it was first published. Runs and grades are never overwritten: a retest or a re-grade publishes a new result and keeps the earlier one readable.

from the publication record
11 November 2026First publishedConverting a complex PDF into clean Markdown with a hosted API v1

Act on this result

Nothing filed here edits the run or the grade. A challenge opens a review, and a review can produce a new run or a re-grade — which becomes the current result and leaves this one in the history.

This matches what I see

You run the same kind of test against your own setup and get the same behaviour.

Agree →
This does not match

Yours behaves differently. Tell us what you got, with a screenshot if you have one.

Disagree →
Point out an issue

Something here is wrong — a reference value, a transcription, a grade.

Report an issue →
Request a retest

On a newer build, a larger dataset, or your own setup.

Request a retest →
We have fixed this

Tell us what changed and we schedule a rerun of the failing test case. The old result stays as history.

Vendor notice →
Filed against this evidenceNothing yet. Challenges, counter-evidence and fix notices appear here with their outcome, and stay on the page after they are resolved.
Cite this result
aidemos.com/benchmarks/pdf-to-markdown-apis/results/firecrawl/a-page-laid-out-in-multiple-columns · 0 pass · 1 fail · coverage 1/1 · graded 2026-10-18

The same record is available as structured data through the AI Demos MCP server, with the counts, the coverage and every per-test-case reason carried as fields.

Verify the proof files

These files support this result. Open a file to inspect the original evidence.

Input · Document (PDF) ↗application/pdf · 143 KB
Proof 1 · Screenshot ↗image/png · 848 KB
Proof 2 · Screenshot ↗image/png · 874 KB
Proof 3 · Screenshot ↗image/png · 713 KB
Output file (Markdown) ↗text/markdown · 12 KB
Output file (JSON) ↗application/json · 13 KB
Proof 4 · Screenshot ↗image/png · 88 KB
File fingerprints (SHA-256)

A fingerprint identifies the exact file used for this result.

Input · Document (PDF)4c7e1f83e8fd4525a7a1c935526668f603a51bacf08202074cfa68c76f4afa69
Proof 1 · Screenshotfac3033d2d85fa4c18b41508c6d6811d2d8fe2523d9fd7245fdff18709337ba1
Proof 2 · Screenshot10f21f537dede97347ad891c5c646097a63c851c1c6631ea0a30e46fe82edf5b
Proof 3 · Screenshot6593371bc7547e21c22d69301f3b1a59d0084e0665e376b7fc5dd85617e6e475
Output file (Markdown)c458ed4eab7482c6adcfb190da2dd727547ed87db16c2758d552439c5a2108dc
Output file (JSON)a07682febc87dc56d8ac6281ece232473f5c3a92bd0c791e7e8091a241967bdd
Proof 4 · Screenshot5034f26247155ac0455b2e2f1c3392c891178b4fe6c2a09f8367f628e8a699a5