It did not clarify the ambiguous 'best customers' wording; it silently chose spend-based ranking and ignored the order-frequency part of the question.
What was measured
Ambiguity Handling
Does it clarify unclear business terms instead of guessing silently?
decisive for this rankingtransformation
A database assistant must clarify unclear business terms instead of guessing, or it will return wrong answers. (3 of 3 judges)
What was given, what came back
Test input: Best customers with unpaid-order and payment-method follow-ups · text · group: live-database-plain-english-queries
Input — what we sent
The exact prompt
Who are my best customers — the ones who order the most and spend the most? Follow-up 1: For the top 3 from that list — do any of them have unpaid orders? Follow-up 2: What payment methods do these top 3 usually use?
A conversational multi-table customer analysis with follow-up questions. It asks for the best customers by both order volume and spend, then drills into unpaid orders for the top 3 and their usual payment methods. Designed to test ranking logic, join-heavy analysis, and follow-up context retention.
Why this input is hard
- · ambiguous business term interpretation
- · multi-table joins
- · aggregation and ranking
- · follow-up context retention
- · scoping to a selected subset
- · payment behavior analysis
Output — unretouched
Also checked on this input — same tool, 4 other criteria
Business Insight✓ WorkedIt adds useful interpretation by flagging Rahul Sharma's unpaid $2,199 order, noting it had been confirmed but unpaid since April 2025, and linking his payment methods to the issue as likely an oversight rather than a pattern.Follow-Up Context✓ WorkedIt retained the top-3 customer context across both follow-ups, answering unpaid-order and payment-method questions for the same three customers from the initial list.Plain English Query Handling◐ MixedIt accepted the natural-language request but only ranked customers by total spend, so it did not fully understand the combined 'order the most and spend the most' intent.Result Readability✓ WorkedIt returns a ranked table with order counts, total spend, and average order value that is straightforward to read and scan.
Provenance
- Observation
- 0c9c2717-901f-4b0f-a114-233eda2981c1
- Evidence run
- af2abc96-3311-484b-a19d-854a2fdd2bf3
- Study
- Query Live Databases Using Plain English with AI
- Research task
- 86b9y6c99
- Tested at
- not recorded
- Source
- first-party
- Evidence state
- verified
- Proof shown
- input + output shown
- Cost / latency
- not captured
- Repeat run
- not captured
- Tester
- not captured
The last three rows are honest blanks, not placeholders — our capture has no field for them yet.
Query this
get_evidence({
tool: "definite",
scenario: "live-database-plain-english-queries"
})MCP · mcp.aidemos.com/api/mcp
Free with attribution.
Same input, same check — 10 other tools
measured on Ambiguity Handling
AI for Database✓ WorkedIt resolves the 'order the most AND spend the most' ambiguity by showing both rankings side by side instead of silently collapsing the request into one metric, and it names the customer who tops both lists.Anomaly AI✓ WorkedWhen 'best customers' was ambiguous, it did not guess silently; it exposed a two-axis ranking, stated the cancelled-order and total_amount assumptions, and separately ranked repeat business versus spend.AskYourDatabase✓ WorkedIt handles the ambiguous 'best customers' request by surfacing both order-count and spend rankings instead of silently choosing one metric.Basedash⚠ StruggledIt did not ask what 'top 3 from that list' meant; instead it silently chose the spend ranking first and then added the order-count check.BlazeSQL✓ WorkedIt clarifies the ambiguous phrase 'best customers' by detecting that cancelled and failed orders could skew the ranking, then rerunning the analysis on paid, non-cancelled orders instead of silently keeping the first pass.camelAI✓ WorkedIt resolved the 'order the most and spend the most' ambiguity by returning separate frequency and spend rankings plus a merged best-overall verdict naming Rahul Sharma as the strongest combination of frequency and value.Dot✓ WorkedFaced with 'order the most and spend the most,' it did not silently choose one dimension; it built a combined equal-weight rank and explicitly surfaced the trade-off between all-around value and single-big-order spend.Draxlr✓ WorkedIt resolved the vague phrase best customers as highest order count plus highest total spend instead of asking for a definition.FutureSmart NL2SQL Agent◐ MixedIt states a ranking heuristic in prose, using spend as the primary sort and order count as a secondary sort, but it still leaves the no-status-filter assumption unstated and does not ask the user to clarify the two-dimensional wording.Querio✓ WorkedResolves the ambiguous phrase 'best customers' by answering with both top-by-orders and top-by-spend views rather than silently choosing one.
This evidence is published in
From the same study (page rebuilt from a later run)
Real inputs and real outputs, no retouching · every cell queryable via API & MCP · aidemos.com