AI Demos Research — the structured-intelligence platform. Every verdict on these pages opens to the execution behind it.
Graded 13 November 2026

Does camelAI ask what an ambiguous follow-up means before answering?

camelAI answered the ambiguous follow-up without clarifying its meaning. It compared the customer rankings by order count and spending, describing where the lists overlapped and how purchasing patterns differed.

1 of 1 test case failed

Every test case in this scenario has a result.

Pass rate0%0 of 1 gradable
Coverage1 of 1test case with a result
0PassNo test case passed.
1FailMissed at least one thing it was expected to do; the reason and proof are with each test case below.
0Not gradableEvery result here could be graded.
0UntestedEvery test case in this scenario has a result.
The pass rate is a summary. The evidence is the test case below: what we sent, what we checked, what the tool returned, and the proof.

The test case

Each test case is judged on its own: Pass, Fail, Not gradable, or Untested. The scenario result above is based on this test case.

"Compare them" with two candidate setsFailEvidence
What we sent

Input 1 · Initial comparison request: “Compare our top 3 customers by order count against our top 3 customers by total spend.”

Input 2 · Ambiguous follow-up: “Compare them.”

What the tool returned

Initial answer, continued

Comparison: Dev is the clear standout, ranking near the top in order count and first in spend. Rosa also appears in both groups, tied at the order-count cutoff. Priya makes the spend top three despite fewer orders, while Wei and Jae-won have high order counts but lower spend. Spend is calculated as order totals less returns marked refunded; cancelled orders are excluded. The connected database records USD orders from Oct 15, 2025 through Oct 21, 2026.

Answer to the ambiguous follow-up

The lists overlap on Dev Raghavan and Rosa Delgado (Rosa is tied at the order-count cutoff). • Dev is strong on both: 20 orders and $20,221.80 net spend—the highest spend. • Rosa also ranks high in both: 18 orders and $13,903.79 net spend. • Wei Zhang has 20 orders but $11,033.29 spend, so their orders average less. • Jae-won Park ties at 18 orders, with $13,646.96 spend. • Priya Raghavan has only 17 orders but $17,202.51 spend—second highest, at about $1,012 per order. In short, order frequency and total spend don't line up perfectly: Priya spends more per order, while Wei's high order count doesn't translate to a top-three spend total. The 18-order tie means four customers qualify at the order-count cutoff.

Transcribed from Proof 2 · Screenshot

Every message the tool sent, in order (3)
  1. Initial answer: customer rankings

    Top customers by order count Among non-cancelled orders; there's a tie at the third-place cutoff. Customer | Orders | Net spend Wei Zhang | 20 | $11,033.29 Dev Raghavan | 20 | $20,221.80 Jae-won Park | 18 | $13,646.96 Rosa Delgado | 18 | $13,903.79 Top customers by total spend Rank | Customer | Net spend | Orders 1 | Dev Raghavan | $20,221.80 | 20 2 | Priya Raghavan | $17,202.51 | 17 3 | Rosa Delgado | $13,903.79 | 18

  2. Initial answer, continued

    Printed above.

  3. Answer to the ambiguous follow-up

    The final response, printed above.

Expected vs. Found
✗Expected: Ask which comparison the customer intends before answering an ambiguous follow-up.Found: It immediately compared the lists without asking a clarifying question.
Supporting proof
Proof 1Screenshot
Chat screenshot: the initial question and customer rankings by order count and spending
Open original ↗
Proof 2Screenshot
Chat screenshot: the initial answer’s closing paragraphs and the follow-up comparison
Open original ↗
Proof 3Screen recording
Screen recording: customer rankings, the follow-up comparison and the expanded work pane
Key moments
Open original ↗
Proof 4Screenshot
Work-pane screenshot: a single reasoning step and no new database query
Open original ↗
Proof 5Screenshot
Database query screenshot: reference customer rankings by order count and spending
Open original ↗
Why this result

Two customer lists made the follow-up genuinely ambiguous. Asking which comparison was intended was required; choosing one silently did not meet that requirement.

Also observed in this test case

Tied customer omitted. The initial order-count table showed two customers tied at 18 orders; the database comparison showed three, including a tied customer the table omitted.

Tie acknowledged later. The follow-up answer’s closing paragraph said four customers qualified at the 18-order cutoff, acknowledging the tie after starting the comparison.

Tested by Anchal Rana · evidence dated 27 October 2026

Configuration and setup

How this tool was set up for the run and what the test needed in place. Each row is a fact from the run's records; a fact the records do not hold is left out, not guessed.

Software that produced the output
camelAI
Build or version
Not exposed by the hosted product · model camelCode.
Plan or tier
Free
Surface
Web app
Set up before the run
The initial answer needed to offer two equally plausible referents for the follow-up: customer rankings by order count and by total spending.
Tested
27 October 2026 · Anchal Rana

How this scenario is graded

How we decide Pass, Fail and Not gradable. The same rules apply to every tool tested on this scenario.

How results are decided

Each test case gets one result per tool: Pass, Fail or Not gradable. A test case we haven't run yet shows Untested. There are no partial results.

The rules
  • Pass: the tool did everything the test expected, and nothing it said contradicts the correct answer.
  • Fail: at least one expected behaviour clearly didn't happen; the test case's result says which one and quotes the tool.
  • Not gradable: our evidence couldn't settle the outcome (for example a record we needed is missing). It is never counted as a fail, and the test case's result says what's missing.

Where this sits in the benchmark

This page is one cell of a larger study: one tool, one scenario. Only this benchmark's frame appears here.

LevelNameScope
BenchmarkAI Database Agents →28 scenarios · 11 tools
CapabilityConversational Interaction →
ScenarioThe follow-up is ambiguous →
ToolcamelAI →

History of this result

What has happened to this result since it was first published. Runs and grades are never overwritten: a retest or a re-grade publishes a new result and keeps the earlier one readable.

from the publication record
13 November 2026First publishedAI Database Agents v1

Act on this result

Nothing filed here edits the run or the grade. A challenge opens a review, and a review can produce a new run or a re-grade — which becomes the current result and leaves this one in the history.

This matches what I see

You run the same kind of test against your own setup and get the same behaviour.

Agree →
This does not match

Yours behaves differently. Tell us what you got, with a screenshot if you have one.

Disagree →
Point out an issue

Something here is wrong — a reference value, a transcription, a grade.

Report an issue →
Request a retest

On a newer build, a larger dataset, or your own setup.

Request a retest →
We have fixed this

Tell us what changed and we schedule a rerun of the failing test case. The old result stays as history.

Vendor notice →
Filed against this evidenceNothing yet. Challenges, counter-evidence and fix notices appear here with their outcome, and stay on the page after they are resolved.
Cite this result
aidemos.com/benchmarks/ai-database-agents/results/camelai/the-follow-up-is-ambiguous · 0 pass · 1 fail · coverage 1/1 · graded 2026-11-13

The same record is available as structured data through the AI Demos MCP server, with the counts, the coverage and every per-test-case reason carried as fields.

Verify the proof files

These files support this result. Open a file to inspect the original evidence.

Proof 1 · Screenshot ↗image/png · 312 KB
Proof 2 · Screenshot ↗image/png · 427 KB
Proof 4 · Screenshot ↗image/png · 405 KB
Proof 5 · Screenshot ↗image/png · 208 KB
File fingerprints (SHA-256)

A fingerprint identifies the exact file used for this result.

Proof 1 · Screenshot1c67589cfca58d7c9c6499279ec0e15be62a71e3e9a3c9d927441fd75f2dc3da
Proof 2 · Screenshote16df6d42cfb84e176c94a3a078f827b57496b0fc8a4630e13356efe89a0bd72
Proof 3 · Screen recordingd0120cdd88746ef8151287bfcc2a616574dde9e8c5411d2729d92802b6fa3c40
Proof 4 · Screenshote8dc7bdd4390430648029dabb9f58b0067c4e88479c7ace048a4d7c6b16c62b6
Proof 5 · Screenshot0959dedb9decb398ace479c189e2ff5cc5cfbceef833053476898280ceb38d14