Graded 7 November 2026
Does Mistral AI keep a code block formatted in Markdown?
Mistral AI returned code without fences and flattened its nested indentation when converting the PDF to Markdown. The Python code was no longer runnable, and the shell command lost its continuation indentation.
1 of 1 test case failed
Every test case in this scenario has a result.
Pass rate0%0 of 1 gradable
Coverage1 of 1test case with a result
0PassNo test case passed.
1FailMissed at least one thing it was expected to do; the reason and proof are with each test case below.
0Not gradableEvery result here could be graded.
0UntestedEvery test case in this scenario has a result.
The pass rate is a summary. The evidence is the test case below: what we sent, what we checked, what the tool returned, and the proof.
The test case
Each test case is judged on its own: Pass, Fail, Not gradable, or Untested. The scenario result above is based on this test case.
Code block — fenced, structure intactFailEvidence
What we sent
InputDocument (PDF)
What the tool returned
Markdownthe whole file · copied lines 1–102 marked
Tarnbeck Institute of Hydrology · Data Services # Reloading a station export after a gauge reset Operations note DS/OP/07 · Data services · Reviewed 14 January 2026 ## Symptoms The station's daily total is short and the loader's own log says nothing is wrong. In the log the day looks like a clean run: the export was fetched, every block was read, and the summary line reports fewer readings than the station's counter did. The gap is always at the end of the day rather than the beginning, and it is always a whole number of blocks. The other sign is a duplicate warning in the block log, with two entries carrying the same sequence number and different checksums. One warning is normal after a reset. A run of them means the reset happened part way through the day and everything after it is being rejected block by block. ## Why the loader rejects the second block Block sequence numbers come from the station and not from the loader. The loader keeps the highest number it has committed for the day and treats anything at or below it as a repeat, which is the right thing to do on a poor link: a station that retransmits sends the same block twice, and the loader must not count it twice. A reset returns the counter to zero, so the blocks after it carry sequence numbers the loader has already committed. Nothing about them looks new and the loader discards them for exactly the reason it should. The reset time is the only thing that distinguishes a continuation from a repeat, and the loader does not know it unless it is told. ## The fix When a gauge is reset in the field the station's counter returns to zero, and the export for that day contains two blocks carrying the same sequence number. The loader accepts the first and rejects the second. That is the right behaviour for a duplicated transmission and the wrong behaviour here, and the day's record ends up short by however much arrived after the reset. The fix is to requeue the affected blocks with the reset time supplied, so that the loader treats the second block as a continuation rather than a repeat. The function below is the one to call. It lives in `tarnbeck.stations.maintenance` and is safe to run more than once: a block that already carries a good checksum is skipped. DS/OP/07 · Data services Page 1 of 3 Tarnbeck Institute of Hydrology · Data Services DS/OP/07 from tarnbeck.stations import Export, reload_window def reload_after_reset(station, reset_at): export = Export.open(station) window = reload_window(reset_at, margin_minutes=45) for block in export.blocks(window): if block.checksum_ok(): continue export.requeue(block, reason="gauge reset") return export.commit(dry_run=False) Run it from the loader host and not from a workstation, and take the reset time from the field log rather than from the station clock, which is the thing that was wrong. The margin defaults to forty-five minutes either side of the reset; widen it only if the field log is vague about the time, because a wide window requeues blocks that were never in doubt. If commit returns a count lower than the number of blocks in the window, stop there and raise it with data services before running anything again. A short count means blocks were rejected for a second reason, and requeuing them repeatedly will not find out what it was. #### Running it Run it from the loader host, in the maintenance environment, with the station code and the reset time as they appear in the field log: $ ssh loader-01.tarnbeck.example $ tarnbeck-maint reload-after-reset --station VQ1 \ --reset-at '2026-01-29T09:15' --confirm The command prints the window it will use, the number of blocks in it and the number it intends to requeue, and then waits. Read the three numbers before answering. If the window is wider than the field log justifies, stop and narrow it rather than letting it run. Without --confirm the command does everything except commit, which is the safe way to see what it would do. The dry run takes a few seconds and there is no reason to skip it. #### Afterwards Check the day's totals against the gauge's own counter before closing the ticket. The loader reports the number of blocks committed and not the number of readings, and a day can come back with the right number of blocks and the wrong number of readings if a block was truncated in the field. Note the reset in the station log, with the time taken from the field log and not the time the reload ran. The next person to look at a gap in the series will look there first, and a reload with nothing written beside it is indistinguishable from a gap nobody has dealt with. DS/OP/07 · Data services Page 2 of 3 Tarnbeck Institute of Hydrology · Data Services DS/OP/07 ### If the count is short A short count means blocks were rejected for a second reason and the reload has not found out what it was. Do not run it again: a second pass requires the same blocks, reports the same count, and fills the log with attempts that bury the original failure. Take the sequence numbers the command reports as rejected and look them up in the block log. A checksum failure on a block truncated in the field cannot be fixed by reloading and needs the station visited. A rejection with no reason recorded is a loader fault, and goes to data services with the run's log attached. ### The block header, for reference Each block in an export begins with a fixed header, which is what the loader reads to decide whether it has seen the block before: SEQ=00184 STATION=VQ1 START=2026-01-29T08:45Z COUNT=0180 INTERVAL=15s CRC=8f2a41d6 SEQ is the station's counter and is the field a reset returns to zero. START is the station clock, which is not to be trusted after a reset - which is why the reset time comes from the field log and not from the export. CRC covers the readings and not the header, so a block with a good checksum and a wrong sequence number is possible and is exactly what a reset produces. Nothing in the header says a reset happened. ### Rolling back A committed reload can be undone within the same day by reverting the station's day to the snapshot the loader takes before it commits. The snapshot is kept for seven days and is named for the run. After seven days the only route back is a fresh fetch of the export from the station, which is possible while the station's own buffer still holds the day - about three weeks - and impossible afterwards. A reload that has committed and cannot be rolled back is an incident and is raised as one, whatever the size of the gap. DS/OP/07 · Data services Page 3 of 3
Returned Markdown: full document; code has no fences, and nested lines share one indentation level
Copied from Output file (Markdown), lines 1–102
Open the Markdown file ↗Output file (JSON)
Cited values4 values from this file
- model
- mistral-ocr-latest
- usage_info
- { "pages_processed": 3, "doc_size_bytes": 80041 }
- pages 2 › markdown
- Tarnbeck Institute of Hydrology · Data Services DS/OP/07 from tarnbeck.stations import Export, reload_window def reload_after_reset(station, reset_at): export = Export.open(station) window = reload_window(reset_at, margin_minutes=45) for block in export.blocks(window): if block.checksum_ok(): continue export.requeue(block, reason="gauge reset") return export.commit(dry_run=False) Run it from the loader host and not from a workstation, and take the reset time from the field log rather than from the station clock, which is the thing that was wrong. The margin defaults to forty-five minutes either side of the reset; widen it only if the field log is vague about the time, because a wide window requeues blocks that were never in doubt. If commit returns a count lower than the number of blocks in the window, stop there and raise it with data services before running anything again. A short count means blocks were rejected for a second reason, and requeuing them repeatedly will not find out what it was. #### Running it Run it from the loader host, in the maintenance environment, with the station code and the reset time as they appear in the field log: $ ssh loader-01.tarnbeck.example $ tarnbeck-maint reload-after-reset --station VQ1 \ --reset-at '2026-01-29T09:15' --confirm The command prints the window it will use, the number of blocks in it and the number it intends to requeue, and then waits. Read the three numbers before answering. If the window is wider than the field log justifies, stop and narrow it rather than letting it run. Without --confirm the command does everything except commit, which is the safe way to see what it would do. The dry run takes a few seconds and there is no reason to skip it. #### Afterwards Check the day's totals against the gauge's own counter before closing the ticket. The loader reports the number of blocks committed and not the number of readings, and a day can come back with the right number of blocks and the wrong number of readings if a block was truncated in the field. Note the reset in the station log, with the time taken from the field log and not the time the reload ran. The next person to look at a gap in the series will look there first, and a reload with nothing written beside it is indistinguishable from a gap nobody has dealt with. DS/OP/07 · Data services Page 2 of 3
- pages 2 › blocks
- [ { "top_left_x": 79, "top_left_y": 58, "bottom_right_x": 298, "bottom_right_y": 71, "content": "Tarnbeck Institute of Hydrology · Data Services", "confidence_scores": null, "type": "header" }, { "top_left_x": 590, "top_left_y": 58, "bottom_right_x": 639, "bottom_right_y": 70, "content": "DS/OP/07", "confidence_scores": null, "type": "header" }, { "top_left_x": 90, "top_left_y": 98, "bottom_right_x": 426, "bottom_right_y": 297, "content": "from tarnbeck.stations import Export, reload_window\n\ndef reload_after_reset(station, reset_at):\n export = Export.open(station)\n window = reload_window(reset_at, margin_minutes=45)\n\n for block in export.blocks(window):\n if block.checksum_ok():\n continue\n export.requeue(block, reason=\"gauge reset\")\n\n return export.commit(dry_run=False)", "confidence_scores": null, "type": "code" }, { "top_left_x": 78, "top_left_y": 313, "bottom_right_x": 623, "bottom_right_y": 384, "content": "Run it from the loader host and not from a workstation, and take the reset time from the field log rather than from the station clock, which is the thing that was wrong. The margin defaults to forty-five minutes either side of the reset; widen it only if the field log is vague about the time, because a wide window requeues blocks that were never in doubt.", "confidence_scores": null, "type": "text" }, { "top_left_x": 78, "top_left_y": 393, "bottom_right_x": 623, "bottom_right_y": 446, "content": "If commit returns a count lower than the number of blocks in the window, stop there and raise it with data services before running anything again. A short count means blocks were rejected for a second reason, and requeuing them repeatedly will not find out what it was.", "confidence_scores": null, "type": "text" }, { "top_left_x": 79, "top_left_y": 469, "bottom_right_x": 146, "bottom_right_y": 484, "content": "#### Running it", "confidence_scores": null, "type": "title" }, { "top_left_x": 78, "top_left_y": 486, "bottom_right_x": 621, "bottom_right_y": 522, "content": "Run it from the loader host, in the maintenance environment, with the station code and the reset time as they appear in the field log:", "confidence_scores": null, "type": "text" }, { "top_left_x": 89, "top_left_y": 556, "bottom_right_x": 403, "bottom_right_y": 602, "content": "$ ssh loader-01.tarnbeck.example\n$ tarnbeck-maint reload-after-reset --station VQ1 \\\n--reset-at '2026-01-29T09:15' --confirm", "confidence_scores": null, "type": "code" }, { "top_left_x": 78, "top_left_y": 621, "bottom_right_x": 634, "bottom_right_y": 673, "content": "The command prints the window it will use, the number of blocks in it and the number it intends to requeue, and then waits. Read the three numbers before answering. If the window is wider than the field log justifies, stop and narrow it rather than letting it run.", "confidence_scores": null, "type": "text" }, { "top_left_x": 78, "top_left_y": 682, "bottom_right_x": 612, "bottom_right_y": 715, "content": "Without --confirm the command does everything except commit, which is the safe way to see what it would do. The dry run takes a few seconds and there is no reason to skip it.", "confidence_scores": null, "type": "text" }, { "top_left_x": 79, "top_left_y": 738, "bottom_right_x": 151, "bottom_right_y": 752, "content": "#### Afterwards", "confidence_scores": null, "type": "title" }, { "top_left_x": 78, "top_left_y": 758, "bottom_right_x": 634, "bottom_right_y": 810, "content": "Check the day's totals against the gauge's own counter before closing the ticket. The loader reports the number of blocks committed and not the number of readings, and a day can come back with the right number of blocks and the wrong number of readings if a block was truncated in the field.", "confidence_scores": null, "type": "text" }, { "top_left_x": 78, "top_left_y": 819, "bottom_right_x": 630, "bottom_right_y": 871, "content": "Note the reset in the station log, with the time taken from the field log and not the time the reload ran. The next person to look at a gap in the series will look there first, and a reload with nothing written beside it is indistinguishable from a gap nobody has dealt with.", "confidence_scores": null, "type": "text" }, { "top_left_x": 79, "top_left_y": 942, "bottom_right_x": 192, "bottom_right_y": 955, "content": "DS/OP/07 · Data services", "confidence_scores": null, "type": "footer" }, { "top_left_x": 588, "top_left_y": 942, "bottom_right_x": 639, "bottom_right_y": 955, "content": "Page 2 of 3", "confidence_scores": null, "type": "footer" } ]
Show the whole file (24 KB)
JSON
{ "pages": [ { "index": 0, "markdown": "Tarnbeck Institute of Hydrology \u00b7 Data Services\n\n# Reloading a station export after a gauge reset\n\nOperations note DS/OP/07 \u00b7 Data services \u00b7 Reviewed 14 January 2026\n\n## Symptoms\n\nThe station's daily total is short and the loader's own log says nothing is wrong. In the log the day looks like a clean run: the export was fetched, every block was read, and the summary line reports fewer readings than the station's counter did. The gap is always at the end of the day rather than the beginning, and it is always a whole number of blocks.\n\nThe other sign is a duplicate warning in the block log, with two entries carrying the same sequence number and different checksums. One warning is normal after a reset. A run of them means the reset happened part way through the day and everything after it is being rejected block by block.\n\n## Why the loader rejects the second block\n\nBlock sequence numbers come from the station and not from the loader. The loader keeps the highest number it has committed for the day and treats anything at or below it as a repeat, which is the right thing to do on a poor link: a station that retransmits sends the same block twice, and the loader must not count it twice.\n\nA reset returns the counter to zero, so the blocks after it carry sequence numbers the loader has already committed. Nothing about them looks new and the loader discards them for exactly the reason it should. The reset time is the only thing that distinguishes a continuation from a repeat, and the loader does not know it unless it is told.\n\n## The fix\n\nWhen a gauge is reset in the field the station's counter returns to zero, and the export for that day contains two blocks carrying the same sequence number. The loader accepts the first and rejects the second. That is the right behaviour for a duplicated transmission and the wrong behaviour here, and the day's record ends up short by however much arrived after the reset.\n\nThe fix is to requeue the affected blocks with the reset time supplied, so that the loader treats the second block as a continuation rather than a repeat. The function below is the one to call. It lives in `tarnbeck.stations.maintenance` and is safe to run more than once: a block that already carries a good checksum is skipped.\n\nDS/OP/07 \u00b7 Data services\n\nPage 1 of 3", "images": [], "tables": [], "hyperlinks": [], "header": null, "footer": null, "dimensions": { "dpi": 87, "height": 1018, "width": 720 }, "confidence_scores": null, "blocks": [ { "top_left_x": 79, "top_left_y": 70, "bottom_right_x": 332, "bottom_right_y": 86, "content": "Tarnbeck Institute of Hydrology \u00b7 Data Services", "confidence_scores": null, "type": "header" }, { "top_left_x": 79, "top_left_y": 92, "bottom_right_x": 493, "bottom_right_y": 114, "content": "# Reloading a station export after a gauge reset", "confidence_scores": null, "type": "title" }, { "top_left_x": 79, "top_left_y": 122, "bottom_right_x": 430, "bottom_right_y": 138, "content": "Operations note DS/OP/07 \u00b7 Data services \u00b7 Reviewed 14 January 2026", "confidence_scores": null, "type": "text" }, { "top_left_x": 79, "top_left_y": 170, "bottom_right_x": 148, "bottom_right_y": 186, "content": "## Symptoms", "confidence_scores": null, "type": "title" }, { "top_left_x": 79, "top_left_y": 191, "bottom_right_x": 626, "bottom_right_y": 260, "content": "The station's daily total is short and the loader's own log says nothing is wrong. In the log the day looks like a clean run: the export was fetched, every block was read, and the summary line reports fewer readings than the station's counter did. The gap is always at the end of the day rather than the beginning, and it is always a whole number of blocks.", "confidence_scores": null, "type": "text" }, { "top_left_x": 79, "top_left_y": 269, "bottom_right_x": 631, "bottom_right_y": 322, "content": "The other sign is a duplicate warning in the block log, with two entries carrying the same sequence number and different checksums. One warning is normal after a reset. A run of them means the reset happened part way through the day and everything after it is being rejected block by block.", "confidence_scores": null, "type": "text" }, { "top_left_x": 79, "top_left_y": 342, "bottom_right_x": 326, "bottom_right_y": 360, "content": "## Why the loader rejects the second block", "confidence_scores": null, "type": "title" }, { "top_left_x": 79, "top_left_y": 363, "bottom_right_x": 635, "bottom_right_y": 432, "content": "Block sequence numbers come from the station and not from the loader. The loader keeps the highest number it has committed for the day and treats anything at or below it as a repeat, which is the right thing to do on a poor link: a station that retransmits sends the same block twice, and the loader must not count it twice.", "confidence_scores": null, "type": "text" }, { "top_left_x": 79, "top_left_y": 442, "bottom_right_x": 618, "bottom_right_y": 512, "content": "A reset returns the counter to zero, so the blocks after it carry sequence numbers the loader has already committed. Nothing about them looks new and the loader discards them for exactly the reason it should. The reset time is the only thing that distinguishes a continuation from a repeat, and the loader does not know it unless it is told.", "confidence_scores": null, "type": "text" }, { "top_left_x": 79, "top_left_y": 533, "bottom_right_x": 126, "bottom_right_y": 549, "content": "## The fix", "confidence_scores": null, "type": "title" }, { "top_left_x": 79, "top_left_y": 554, "bottom_right_x": 640, "bottom_right_y": 624, "content": "When a gauge is reset in the field the station's counter returns to zero, and the export for that day contains two blocks carrying the same sequence number. The loader accepts the first and rejects the second. That is the right behaviour for a duplicated transmission and the wrong behaviour here, and the day's record ends up short by however much arrived after the reset.", "confidence_scores": null, "type": "text" }, { "top_left_x": 79, "top_left_y": 633, "bottom_right_x": 632, "bottom_right_y": 704, "content": "The fix is to requeue the affected blocks with the reset time supplied, so that the loader treats the second block as a continuation rather than a repeat. The function below is the one to call. It lives in `tarnbeck.stations.maintenance` and is safe to run more than once: a block that already carries a good checksum is skipped.", "confidence_scores": null, "type": "text" }, { "top_left_x": 79, "top_left_y": 941, "bottom_right_x": 192, "bottom_right_y": 955, "content": "DS/OP/07 \u00b7 Data services", "confidence_scores": null, "type": "footer" }, { "top_left_x": 587, "top_left_y": 941, "bottom_right_x": 639, "bottom_right_y": 955, "content": "Page 1 of 3", "confidence_scores": null, "type": "footer" } ] }, { "index": 1, "markdown": "Tarnbeck Institute of Hydrology \u00b7 Data Services\n\nDS/OP/07\n\nfrom tarnbeck.stations import Export, reload_window\n\ndef reload_after_reset(station, reset_at):\n export = Export.open(station)\n window = reload_window(reset_at, margin_minutes=45)\n\n for block in export.blocks(window):\n if block.checksum_ok():\n continue\n export.requeue(block, reason=\"gauge reset\")\n\n return export.commit(dry_run=False)\n\nRun it from the loader host and not from a workstation, and take the reset time from the field log rather than from the station clock, which is the thing that was wrong. The margin defaults to forty-five minutes either side of the reset; widen it only if the field log is vague about the time, because a wide window requeues blocks that were never in doubt.\n\nIf commit returns a count lower than the number of blocks in the window, stop there and raise it with data services before running anything again. A short count means blocks were rejected for a second reason, and requeuing them repeatedly will not find out what it was.\n\n#### Running it\n\nRun it from the loader host, in the maintenance environment, with the station code and the reset time as they appear in the field log:\n\n$ ssh loader-01.tarnbeck.example\n$ tarnbeck-maint reload-after-reset --station VQ1 \\\n--reset-at '2026-01-29T09:15' --confirm\n\nThe command prints the window it will use, the number of blocks in it and the number it intends to requeue, and then waits. Read the three numbers before answering. If the window is wider than the field log justifies, stop and narrow it rather than letting it run.\n\nWithout --confirm the command does everything except commit, which is the safe way to see what it would do. The dry run takes a few seconds and there is no reason to skip it.\n\n#### Afterwards\n\nCheck the day's totals against the gauge's own counter before closing the ticket. The loader reports the number of blocks committed and not the number of readings, and a day can come back with the right number of blocks and the wrong number of readings if a block was truncated in the field.\n\nNote the reset in the station log, with the time taken from the field log and not the time the reload ran. The next person to look at a gap in the series will look there first, and a reload with nothing written beside it is indistinguishable from a gap nobody has dealt with.\n\nDS/OP/07 \u00b7 Data services\n\nPage 2 of 3", "images": [], "tables": [], "hyperlinks": [], "header": null, "footer": null, "dimensions": { "dpi": 87, "height": 1018, "width": 720 }, "confidence_scores": null, "blocks": [ { "top_left_x": 79, "top_left_y": 58, "bottom_right_x": 298, "bottom_right_y": 71, "content": "Tarnbeck Institute of Hydrology \u00b7 Data Services", "confidence_scores": null, "type": "header" }, { "top_left_x": 590, "top_left_y": 58, "bottom_right_x": 639, "bottom_right_y": 70, "content": "DS/OP/07", "confidence_scores": null, "type": "header" }, { "top_left_x": 90, "top_left_y": 98, "bottom_right_x": 426, "bottom_right_y": 297, "content": "from tarnbeck.stations import Export, reload_window\n\ndef reload_after_reset(station, reset_at):\n export = Export.open(station)\n window = reload_window(reset_at, margin_minutes=45)\n\n for block in export.blocks(window):\n if block.checksum_ok():\n continue\n export.requeue(block, reason=\"gauge reset\")\n\n return export.commit(dry_run=False)", "confidence_scores": null, "type": "code" }, { "top_left_x": 78, "top_left_y": 313, "bottom_right_x": 623, "bottom_right_y": 384, "content": "Run it from the loader host and not from a workstation, and take the reset time from the field log rather than from the station clock, which is the thing that was wrong. The margin defaults to forty-five minutes either side of the reset; widen it only if the field log is vague about the time, because a wide window requeues blocks that were never in doubt.", "confidence_scores": null, "type": "text" }, { "top_left_x": 78, "top_left_y": 393, "bottom_right_x": 623, "bottom_right_y": 446, "content": "If commit returns a count lower than the number of blocks in the window, stop there and raise it with data services before running anything again. A short count means blocks were rejected for a second reason, and requeuing them repeatedly will not find out what it was.", "confidence_scores": null, "type": "text" }, { "top_left_x": 79, "top_left_y": 469, "bottom_right_x": 146, "bottom_right_y": 484, "content": "#### Running it", "confidence_scores": null, "type": "title" }, { "top_left_x": 78, "top_left_y": 486, "bottom_right_x": 621, "bottom_right_y": 522, "content": "Run it from the loader host, in the maintenance environment, with the station code and the reset time as they appear in the field log:", "confidence_scores": null, "type": "text" }, { "top_left_x": 89, "top_left_y": 556, "bottom_right_x": 403, "bottom_right_y": 602, "content": "$ ssh loader-01.tarnbeck.example\n$ tarnbeck-maint reload-after-reset --station VQ1 \\\n--reset-at '2026-01-29T09:15' --confirm", "confidence_scores": null, "type": "code" }, { "top_left_x": 78, "top_left_y": 621, "bottom_right_x": 634, "bottom_right_y": 673, "content": "The command prints the window it will use, the number of blocks in it and the number it intends to requeue, and then waits. Read the three numbers before answering. If the window is wider than the field log justifies, stop and narrow it rather than letting it run.", "confidence_scores": null, "type": "text" }, { "top_left_x": 78, "top_left_y": 682, "bottom_right_x": 612, "bottom_right_y": 715, "content": "Without --confirm the command does everything except commit, which is the safe way to see what it would do. The dry run takes a few seconds and there is no reason to skip it.", "confidence_scores": null, "type": "text" }, { "top_left_x": 79, "top_left_y": 738, "bottom_right_x": 151, "bottom_right_y": 752, "content": "#### Afterwards", "confidence_scores": null, "type": "title" }, { "top_left_x": 78, "top_left_y": 758, "bottom_right_x": 634, "bottom_right_y": 810, "content": "Check the day's totals against the gauge's own counter before closing the ticket. The loader reports the number of blocks committed and not the number of readings, and a day can come back with the right number of blocks and the wrong number of readings if a block was truncated in the field.", "confidence_scores": null, "type": "text" }, { "top_left_x": 78, "top_left_y": 819, "bottom_right_x": 630, "bottom_right_y": 871, "content": "Note the reset in the station log, with the time taken from the field log and not the time the reload ran. The next person to look at a gap in the series will look there first, and a reload with nothing written beside it is indistinguishable from a gap nobody has dealt with.", "confidence_scores": null, "type": "text" }, { "top_left_x": 79, "top_left_y": 942, "bottom_right_x": 192, "bottom_right_y": 955, "content": "DS/OP/07 \u00b7 Data services", "confidence_scores": null, "type": "footer" }, { "top_left_x": 588, "top_left_y": 942, "bottom_right_x": 639, "bottom_right_y": 955, "content": "Page 2 of 3", "confidence_scores": null, "type": "footer" } ] }, { "index": 2, "markdown": "Tarnbeck Institute of Hydrology \u00b7 Data Services\n\nDS/OP/07\n\n### If the count is short\n\nA short count means blocks were rejected for a second reason and the reload has not found out what it was. Do not run it again: a second pass requires the same blocks, reports the same count, and fills the log with attempts that bury the original failure.\n\nTake the sequence numbers the command reports as rejected and look them up in the block log. A checksum failure on a block truncated in the field cannot be fixed by reloading and needs the station visited. A rejection with no reason recorded is a loader fault, and goes to data services with the run's log attached.\n\n### The block header, for reference\n\nEach block in an export begins with a fixed header, which is what the loader reads to decide whether it has seen the block before:\n\n\nSEQ=00184 STATION=VQ1 START=2026-01-29T08:45Z\nCOUNT=0180 INTERVAL=15s CRC=8f2a41d6\n\n\nSEQ is the station's counter and is the field a reset returns to zero. START is the station clock, which is not to be trusted after a reset - which is why the reset time comes from the field log and not from the export.\n\nCRC covers the readings and not the header, so a block with a good checksum and a wrong sequence number is possible and is exactly what a reset produces. Nothing in the header says a reset happened.\n\n### Rolling back\n\nA committed reload can be undone within the same day by reverting the station's day to the snapshot the loader takes before it commits. The snapshot is kept for seven days and is named for the run.\n\nAfter seven days the only route back is a fresh fetch of the export from the station, which is possible while the station's own buffer still holds the day - about three weeks - and impossible afterwards. A reload that has committed and cannot be rolled back is an incident and is raised as one, whatever the size of the gap.\n\nDS/OP/07 \u00b7 Data services\n\nPage 3 of 3", "images": [], "tables": [], "hyperlinks": [], "header": null, "footer": null, "dimensions": { "dpi": 87, "height": 1018, "width": 720 }, "confidence_scores": null, "blocks": [ { "top_left_x": 79, "top_left_y": 57, "bottom_right_x": 298, "bottom_right_y": 71, "content": "Tarnbeck Institute of Hydrology \u00b7 Data Services", "confidence_scores": null, "type": "header" }, { "top_left_x": 587, "top_left_y": 57, "bottom_right_x": 639, "bottom_right_y": 70, "content": "DS/OP/07", "confidence_scores": null, "type": "header" }, { "top_left_x": 79, "top_left_y": 89, "bottom_right_x": 203, "bottom_right_y": 105, "content": "### If the count is short", "confidence_scores": null, "type": "title" }, { "top_left_x": 79, "top_left_y": 107, "bottom_right_x": 626, "bottom_right_y": 161, "content": "A short count means blocks were rejected for a second reason and the reload has not found out what it was. Do not run it again: a second pass requires the same blocks, reports the same count, and fills the log with attempts that bury the original failure.", "confidence_scores": null, "type": "text" }, { "top_left_x": 79, "top_left_y": 169, "bottom_right_x": 633, "bottom_right_y": 239, "content": "Take the sequence numbers the command reports as rejected and look them up in the block log. A checksum failure on a block truncated in the field cannot be fixed by reloading and needs the station visited. A rejection with no reason recorded is a loader fault, and goes to data services with the run's log attached.", "confidence_scores": null, "type": "text" }, { "top_left_x": 79, "top_left_y": 261, "bottom_right_x": 272, "bottom_right_y": 277, "content": "### The block header, for reference", "confidence_scores": null, "type": "title" }, { "top_left_x": 79, "top_left_y": 280, "bottom_right_x": 592, "bottom_right_y": 314, "content": "Each block in an export begins with a fixed header, which is what the loader reads to decide whether it has seen the block before:", "confidence_scores": null, "type": "text" }, { "top_left_x": 91, "top_left_y": 352, "bottom_right_x": 368, "bottom_right_y": 381, "content": "\nSEQ=00184 STATION=VQ1 START=2026-01-29T08:45Z\nCOUNT=0180 INTERVAL=15s CRC=8f2a41d6\n", "confidence_scores": null, "type": "text" }, { "top_left_x": 79, "top_left_y": 398, "bottom_right_x": 638, "bottom_right_y": 451, "content": "SEQ is the station's counter and is the field a reset returns to zero. START is the station clock, which is not to be trusted after a reset - which is why the reset time comes from the field log and not from the export.", "confidence_scores": null, "type": "text" }, { "top_left_x": 79, "top_left_y": 461, "bottom_right_x": 613, "bottom_right_y": 513, "content": "CRC covers the readings and not the header, so a block with a good checksum and a wrong sequence number is possible and is exactly what a reset produces. Nothing in the header says a reset happened.", "confidence_scores": null, "type": "text" }, { "top_left_x": 79, "top_left_y": 533, "bottom_right_x": 159, "bottom_right_y": 550, "content": "### Rolling back", "confidence_scores": null, "type": "title" }, { "top_left_x": 79, "top_left_y": 553, "bottom_right_x": 629, "bottom_right_y": 605, "content": "A committed reload can be undone within the same day by reverting the station's day to the snapshot the loader takes before it commits. The snapshot is kept for seven days and is named for the run.", "confidence_scores": null, "type": "text" }, { "top_left_x": 79, "top_left_y": 615, "bottom_right_x": 638, "bottom_right_y": 686, "content": "After seven days the only route back is a fresh fetch of the export from the station, which is possible while the station's own buffer still holds the day - about three weeks - and impossible afterwards. A reload that has committed and cannot be rolled back is an incident and is raised as one, whatever the size of the gap.", "confidence_scores": null, "type": "text" }, { "top_left_x": 79, "top_left_y": 940, "bottom_right_x": 192, "bottom_right_y": 955, "content": "DS/OP/07 \u00b7 Data services", "confidence_scores": null, "type": "footer" }, { "top_left_x": 585, "top_left_y": 940, "bottom_right_x": 639, "bottom_right_y": 955, "content": "Page 3 of 3", "confidence_scores": null, "type": "footer" } ] } ], "model": "mistral-ocr-latest", "document_annotation": null, "usage_info": { "pages_processed": 3, "doc_size_bytes": 80041 } }
Expected vs. Found
| ✗ | Expected: Code must appear in a preformatted or fenced block, preserving line breaks and indentation.Found: The source had four distinct indentation levels; the returned Python loop had one. Neither code region was fenced, and the shell continuation lost its indentation. |
Supporting proof
Proof 1Screenshot
Why this result
Code needed to retain block formatting and nesting. The complete PDF was processed, but Mistral’s own response already lacked fences and flattened the loop’s nesting. This is a formatting failure in the returned code, not an interrupted conversion.
Also observed in this test case
Code regions recognised. The API labelled both the Python function and the shell commands on page 2 as code.
Code characters preserved. In the page-2 Python function and shell commands, text characters were preserved exactly, including straight quotes and plain hyphens.
Tested by Chandresh Bisht · evidence dated 10 September 2026
Configuration and setup
How this tool was set up for the run and what the test needed in place. Each row is a fact from the run's records; a fact the records do not hold is left out, not guessed.
Software that produced the output
Mistral AI
Build or version
mistral-ocr-latest (model id echoed by the API; dated version not exposed)
Plan or tier
Mistral La Plateforme (metered OCR endpoint)
Surface
API
Set up before the run
The complete maintenance-note PDF, including its nested Python function and indented shell continuation, had to be converted to Markdown in one pass through the default pipeline, without cropped pages or tuned settings.
Pipeline
single pass
Tested
10 September 2026 · Chandresh Bisht
Connected to
How this scenario is graded
How we decide Pass, Fail and Not gradable. The same rules apply to every tool tested on this scenario.
How results are decided
Each test case gets one result per tool: Pass, Fail or Not gradable. A test case we haven't run yet shows Untested. There are no partial results.
The rules
- Pass: the tool did everything the test expected, and nothing it said contradicts the correct answer.
- Fail: at least one expected behaviour clearly didn't happen; the test case's result says which one and quotes the tool.
- Not gradable: our evidence couldn't settle the outcome (for example a record we needed is missing). It is never counted as a fail, and the test case's result says what's missing.
Where this sits in the benchmark
This page is one cell of a larger study: one tool, one scenario. Only this benchmark's frame appears here.
| Level | Name | Scope |
|---|---|---|
| Benchmark | Converting a complex PDF into clean Markdown with a hosted API → | 12 scenarios · 13 tools |
| Capability | Code Extraction → | |
| Scenario | A document containing a code block → | |
| Tool | Mistral AI → |
Global scenario definition → · Global capability definition → · Mistral AI product page →
History of this result
What has happened to this result since it was first published. Runs and grades are never overwritten: a retest or a re-grade publishes a new result and keeps the earlier one readable.
4 November 2026First publishedConverting a complex PDF into clean Markdown with a hosted API v1
11 November 2026Updated — retested
The result is unchanged: 0 pass · 1 fail. The earlier result stays readable. Read the earlier result →
Act on this result
Nothing filed here edits the run or the grade. A challenge opens a review, and a review can produce a new run or a re-grade — which becomes the current result and leaves this one in the history.
This matches what I see
You run the same kind of test against your own setup and get the same behaviour.
Agree →This does not match
Yours behaves differently. Tell us what you got, with a screenshot if you have one.
Disagree →Point out an issue
Something here is wrong — a reference value, a transcription, a grade.
Report an issue →We have fixed this
Tell us what changed and we schedule a rerun of the failing test case. The old result stays as history.
Vendor notice →Filed against this evidenceNothing yet. Challenges, counter-evidence and fix notices appear here with their outcome, and stay on the page after they are resolved.
Cite this result
aidemos.com/benchmarks/pdf-to-markdown-apis/results/mistral-ai/a-document-containing-a-code-block · 0 pass · 1 fail · coverage 1/1 · graded 2026-11-07
The same record is available as structured data through the AI Demos MCP server, with the counts, the coverage and every per-test-case reason carried as fields.
Verify the proof files
These files support this result. Open a file to inspect the original evidence.
Input · Document (PDF) ↗application/pdf · 78 KB
Output file (Markdown) ↗text/markdown; charset=utf-8 · 6 KB
Output file (JSON) ↗application/json · 24 KB
Proof 1 · Screenshot ↗image/png · 67 KB
File fingerprints (SHA-256)
A fingerprint identifies the exact file used for this result.
Input · Document (PDF)cd27d1e77ec9818ff769bde1b49d95e711373888507af4b007054c27b7a52db8
Output file (Markdown)5a64fca7755a8bc54ce041be4a2b1a2b94e3bca1bfc849c435d643c5b6b51ff2
Output file (JSON)3e56354d1a52162319832bc65a2627e2150314d96485deec7df85472f2496337
Proof 1 · Screenshot123fe9ba4fa3226d7cf02c53d72be1861eb3c58cb4b7e6b20a3962c4603ebd02