AI Demos Research — the structured-intelligence platform. Every verdict on these pages opens to the execution behind it.
Graded 9 October 2026

Can BlazeSQL consistently apply a user's standing rule across future answers?

BlazeSQL kept the standing rule across future questions. After the rule was set once, the later revenue answer returned 51,848.02 and the later order-count answer returned 32, and both generated queries still filtered out cancelled orders.

1 of 1 test case passed

Every test case in this scenario has a result.

Pass rate100%1 of 1 gradable
Coverage1 of 1test case with a result
1PassDid everything it was expected to do.
0FailNo test case failed.
0Not gradableEvery result here could be graded.
0UntestedEvery test case in this scenario has a result.
The pass rate is a summary. The evidence is the test case below: what we sent, what we checked, what the tool returned, and the proof.

The test case

Each test case is judged on its own: Pass, Fail, Not gradable, or Untested. The scenario result above is based on this test case.

Standing exclusion rule on later questionsPassEvidence
What we sent

Input 1: “From now on, exclude cancelled orders from every revenue and order-count calculation unless I say otherwise.”

Input 2: “What's our total revenue this year?”

Input 3: “How many orders did we get last month?”

What the tool returned

Response 1 replies to Input 1.

Response 2 replies to Input 2.

Response 3 replies to Input 3.

Text
Response to input 1
Got it! From now on, I'll always filter out cancelled orders (i.e. status != 'cancelled') in any revenue totals or order counts, unless you specifically ask me to include them.

Tip: To make this rule permanent across sessions, add it to your Knowledge Base via the Train Blaze page so I'll always remember it — even in future conversations.

Anything else you'd like to explore?

SUGGESTED NOTE SAVED BY THE TOOL (21 September 2026, 14:50):
Always exclude cancelled orders (status = 'cancelled') from all revenue and order-count calculations unless the user explicitly requests otherwise.
Table: orders
Column: status

Response to input 2
Generated query: Total Revenue This Year

SELECT
    SUM(total_amount) AS total_revenue
FROM orders
WHERE
    status != 'cancelled'
    AND EXTRACT(YEAR FROM placed_at) = 2026

Query logic:
1. Filter orders placed in 2026 and exclude cancelled orders.
2. Sum total_amount to get total revenue.

total_revenue
51,848.02

Done! Cancelled orders are excluded as requested. Would you like to break this down further — by month, category, or something else?

Response to input 3
Generated query: Order Count Last Month

SELECT
    COUNT(*) AS order_count
FROM orders
WHERE
    status != 'cancelled'
    AND EXTRACT(YEAR FROM placed_at) = 2026
    AND EXTRACT(MONTH FROM placed_at) = 8

Query logic:
1. Filter orders placed in August 2026 and exclude cancelled orders.
2. Count the remaining orders.

order_count
32

There you go — August 2026 order count, cancelled orders excluded! Want to dig deeper, like comparing it to previous months?

Transcript: the three turns, replies, and saved note.

Copied from Captured file (text), lines 12–21

Open the text file ↗

Copied from Captured file (text), lines 30–46

Copied from Captured file (text), lines 55–72

Expected vs. Found
✓Expected: Both later answers keep the standing rule without restating it.Found: The revenue and order-count queries kept the cancelled-order filter and returned 51,848.02 and 32.
Supporting proof
Proof 1Screenshot
Chat screenshot: the standing rule acknowledgment and saved note.
The tool’s response is printed above, under What the tool returned. Show this screenshotOpen original ↗
Proof 2Screenshot
Chat screenshot: revenue this year with cancelled orders excluded.
The tool’s response is printed above, under What the tool returned. Show this screenshotOpen original ↗
Proof 3Screenshot
Chat screenshot: last month's order count with cancelled orders excluded.
The tool’s response is printed above, under What the tool returned. Show this screenshotOpen original ↗
Why this result

The Pass stands because the standing rule was set once and both future answers kept cancelled orders filtered out; the run date made August the correct last month.

Also observed in this test case

Date handling. Both later queries use year and month extraction instead of relative date math, which fits this run date but would not track a later calendar change.

Persistent-rule route. The reply points to Train Blaze and the Knowledge Base as the way to make the rule permanent, which is separate from the session rule here.

Tested by Ayush Ghosh · evidence dated 21 September 2026

Configuration and setup

How this tool was set up for the run and what the test needed in place. Each row is a fact from the run's records; a fact the records do not hold is left out, not guessed.

Software that produced the output
BlazeSQL
Build or version
1.8.93
Surface
Web app
Set up before the run
A class of rows had to be excluded because later answers depended on it, and the product was asked to set that standing rule through its training-related mechanism.
Tested
21 September 2026 · Ayush Ghosh

How this scenario is graded

How we decide Pass, Fail and Not gradable. The same rules apply to every tool tested on this scenario.

How results are decided

Each test case gets one result per tool: Pass, Fail or Not gradable. A test case we haven't run yet shows Untested. There are no partial results.

The rules
  • Pass: the tool did everything the test expected, and nothing it said contradicts the correct answer.
  • Fail: at least one expected behaviour clearly didn't happen; the test case's result says which one and quotes the tool.
  • Not gradable: our evidence couldn't settle the outcome (for example a record we needed is missing). It is never counted as a fail, and the test case's result says what's missing.

Where this sits in the benchmark

This page is one cell of a larger study: one tool, one scenario. Only this benchmark's frame appears here.

LevelNameScope
BenchmarkAI Database Agents →28 scenarios · 11 tools
CapabilityTraining →
ScenarioThe user sets a standing rule for all future answers →
ToolBlazeSQL →

History of this result

What has happened to this result since it was first published. Runs and grades are never overwritten: a retest or a re-grade publishes a new result and keeps the earlier one readable.

from the publication record
12 October 2026First publishedAI Database Agents v1

Act on this result

Nothing filed here edits the run or the grade. A challenge opens a review, and a review can produce a new run or a re-grade — which becomes the current result and leaves this one in the history.

This matches what I see

You run the same kind of test against your own setup and get the same behaviour.

Agree →
This does not match

Yours behaves differently. Tell us what you got, with a screenshot if you have one.

Disagree →
Point out an issue

Something here is wrong — a reference value, a transcription, a grade.

Report an issue →
Request a retest

On a newer build, a larger dataset, or your own setup.

Request a retest →
We have fixed this

Tell us what changed and we schedule a rerun of the failing test case. The old result stays as history.

Vendor notice →
Filed against this evidenceNothing yet. Challenges, counter-evidence and fix notices appear here with their outcome, and stay on the page after they are resolved.
Cite this result
aidemos.com/benchmarks/ai-database-agents/results/blazesql/standing-rule-persistence · 1 pass · 0 fail · coverage 1/1 · graded 2026-10-09

The same record is available as structured data through the AI Demos MCP server, with the counts, the coverage and every per-test-case reason carried as fields.

Verify the proof files

These files support this result. Open a file to inspect the original evidence.

Proof 1 · Screenshot ↗image/png · 73 KB
Proof 2 · Screenshot ↗image/png · 37 KB
Proof 3 · Screenshot ↗image/png · 38 KB
Captured file (text) ↗text/plain; charset=utf-8 · 2 KB
File fingerprints (SHA-256)

A fingerprint identifies the exact file used for this result.

Proof 1 · Screenshot7c36fd72712a1a4fbfc741f2b7abf8c501901b5de70a8933f41dbd672e42f9cb
Proof 2 · Screenshot84491157ef45d85545baa34e0b2f54f0dcdedb10e8a8d5029a31bec4a60c0ba2
Proof 3 · Screenshot2506c5094e9e1b86b624c89faa26b436b242b74d41e6df79299f84058957c93d
Captured file (text)9f332ce9b704bf81c48f2c526a640dabdac5f9d023688771314d16ba745a7517