Can Mistral AI handle an ordinary digital text document without missing, garbling, or inventing content?
Mistral AI did not keep an ordinary digital text document complete and unchanged. In the graded output for briefing_note.pdf, running headers and page numbers were spliced into body text at the cited line ranges, including 33-38, 56-64, 110-118, and 138-146.
1 of 1 test case failed
Every test case in this scenario has a result.
The test case
Each test case is judged on its own: Pass, Fail, Not gradable, or Untested. The scenario result above counts this row.
Ordinary digital text — complete and unchangedFailEvidence
Braithe Estuary Partnership Briefing note BEP/BN/2026/04 # Sediment recharge at Kellow Sands: the 2025/26 placement Prepared by the technical secretariat for the Partnership Board · 12 February 2026 ## Background The Partnership has placed dredged sand on the upper foreshore at Kellow Sands in each of the last three winters, under the standing consent granted in 2021 and the method agreed after the 2023 trial. The purpose is to slow the loss of intertidal in front of the saltmarsh, which the 2017 study put at rather more than a metre a year at the northern end. The material is a by-product of maintenance dredging the Harbour Board carries out anyway. Placing it on the foreshore rather than at the licensed offshore site is cheaper for the Board and useful to the Partnership, and the arrangement has run without difficulty since 2021. ## The consent position The standing consent granted in 2021 runs to March 2029 and covers placement of up to 25,000 cubic metres a winter at the two locations shown in the consent drawing. It carries eleven conditions, of which two bear on the operation: placement is not permitted between 1 March and 31 August, and the Partnership must notify the regulator seven days before a campaign begins. Both conditions were met this winter. The notification for the November start was sent on 28 October and acknowledged, and the January loads fell within the same campaign and needed no further notice. The regulator inspected once, on 4 December, and raised nothing. ## The dredging programme The material comes from the Harbour Board's maintenance dredging, which runs on a three-year cycle: the Vardenne Channel and the approach to the quay every year, the bar every second year, and the berths at Marnhead in the third. The volume available to the Partnership therefore varies with the cycle as well as with the winter, and the largest volumes fall in the years the bar is cut. This was a channel-and-approach year, which is the smaller of the three. The Board's dredger worked twenty-two days in the estuary between October and January against thirty-one in the previous season, and the eight loads placed at Kellow Sands were what the programme could offer rather than what the Partnership asked for. The secretariat does not regard the difference as material at this stage of the trial. ## The winter's placement The Partnership placed 18,400 cubic metres of dredged sand on the upper foreshore at Kellow Sands between 6 November 2025 and 22 January 2026. The material came from maintenance dredging of the Vardenne Channel and was placed in eight loads under the standing consent granted in 2021. This note records what was placed, what it cost, and what the winter surveys Page 1 of 7 Braithe Estuary Partnership have shown so far. Briefing note BEP/BN/2026/04 Placement followed the method agreed after the 2023 trial. Each load was discharged at high water on a rising spring tide, at a single point some forty metres seaward of the saltmarsh edge, and left to redistribute without machinery working on the flat. The intention is that the material moves landward under wave action over two or three winters rather than being spread mechanically, which the trial showed disturbs the buried peat horizon at Thrimby Point. ### **How the material is placed** Each load is brought up on the flood and held off the placement point until high water. Discharge takes between forty minutes and an hour and a quarter depending on the load, and the vessel is off the flat and back in the channel within the hour after. The Partnership's own staff attend every discharge from the shore and record the time, the state of the tide, the wind and anything of note; the records are held with the placement file.
Markdown output of the briefing note, with headers and page numbers retained in body text. From outside the graded run; shown for context, not graded.
Copied from Output file (Markdown), lines 1–42 · an excerpt
Open the whole Markdown file ↗These are scenario-level criteria. Each test case's Expected and Found are listed separately.
- Whether every paragraph is preserved complete and unchanged.
- Whether nothing is silently missing, garbled, or invented.
- Whether the result remains recognisably plain text without unintended changes to the content.
| ✗ | Found: Page headers and page numbers were spliced into body sentences at several page breaks. |
The Markdown kept the source text, but page headers and page numbers were carried into the middle of sentences at several page breaks. Under the text-fidelity check, that is not unchanged output, so the result fails.
Nested bold headings. Headings appear at multiple depths, and many keep bold markup inside the heading text.
Character count mismatch. The run metadata reports 19,083 characters; the stored file is 19,096 bytes.
Configuration and setup
How this tool was set up for the run and what the test needed in place. Each row is a fact from the run's records; a fact the records do not hold is left out, not guessed.
How this scenario is graded
How we decide Pass, Fail and Not gradable. The same rules apply to every tool tested on this scenario.
Each test case gets one result per tool: Pass, Fail or Not gradable. A test case we haven't run yet shows Untested. There are no partial results.
- Pass: the tool did everything the test expected, and nothing it said contradicts the correct answer.
- Fail: at least one expected behaviour clearly didn't happen; the row says which and quotes the tool.
- Not gradable: our evidence couldn't settle the outcome (for example a record we needed is missing). It is never counted as a fail, and the row says what's missing.
Where this sits in the benchmark
This page is one cell of a larger study: one tool, one scenario. Only this benchmark's frame appears here.
| Level | Name | Scope |
|---|---|---|
| Benchmark | Converting a complex PDF into clean Markdown with a hosted API → | 12 scenarios · 13 tools |
| Capability | Text Fidelity → | |
| Scenario | An ordinary digital text document → | |
| Tool | Mistral AI → |
Global scenario definition → · Global capability definition → · Mistral AI product page →
History of this result
What has happened to this result since it was first published. Runs and grades are never overwritten: a retest or a re-grade publishes a new result and keeps the earlier one readable.
Act on this result
Nothing filed here edits the run or the grade. A challenge opens a review, and a review can produce a new run or a re-grade — which becomes the current result and leaves this one in the history.
You run the same kind of test against your own setup and get the same behaviour.
Agree →Yours behaves differently. Tell us what you got, with a screenshot if you have one.
Disagree →Something here is wrong — a reference value, a transcription, a grade.
Report an issue →Tell us what changed and we schedule a rerun of the failing test case. The old result stays as history.
Vendor notice →The same record is available as structured data through the AI Demos MCP server, with the counts, the coverage and every per-test-case reason carried as fields.
Verify the proof files
These files support this result. Open a file to inspect the original evidence.
File fingerprints (SHA-256)
A fingerprint identifies the exact file used for this result.