It remembered enough of the prior answer to check both ranking lists, but it still narrowed the follow-up instead of preserving the user's intended scope cleanly.
What was measured
Follow-Up Context
Does the tool remember previous answers correctly?
context, not decisivetransformation
Remembering prior turns improves workflow, but a tool can still do the core job without strong conversation memory. (2 of 3 judges)
What was given, what came back
Test input: Best customers with unpaid-order and payment-method follow-ups · text · group: live-database-plain-english-queries
Input — what we sent
The exact prompt
Who are my best customers — the ones who order the most and spend the most? Follow-up 1: For the top 3 from that list — do any of them have unpaid orders? Follow-up 2: What payment methods do these top 3 usually use?
A conversational multi-table customer analysis with follow-up questions. It asks for the best customers by both order volume and spend, then drills into unpaid orders for the top 3 and their usual payment methods. Designed to test ranking logic, join-heavy analysis, and follow-up context retention.
Why this input is hard
- · ambiguous business term interpretation
- · multi-table joins
- · aggregation and ranking
- · follow-up context retention
- · scoping to a selected subset
- · payment behavior analysis
Output — unretouched
Also checked on this input — same tool, 5 other criteria
Ambiguity Handling⚠ StruggledIt did not ask what 'top 3 from that list' meant; instead it silently chose the spend ranking first and then added the order-count check.Business Insight✓ WorkedIt explains what the numbers mean by naming Rahul Sharma as the best overall customer, Mohan Vishe as the most frequent, and Deepak Kulkarni as the biggest spender.Plain English Query Handling✓ WorkedIt correctly handled a conversational, multi-part customer question without requiring SQL, producing two rankings plus follow-up answers.Result Readability✓ WorkedThe output stays readable for non-technical users, using compact tables and short summaries with explicit customer names and amounts.SQL Generation✓ WorkedIt successfully generated the ranking queries and follow-up lookups, including the top spenders, top order-count customers, unpaid-order status, and usual payment methods.
Provenance
- Observation
- 56f455ce-aca2-4d40-bfe0-67dd5955602e
- Evidence run
- af2abc96-3311-484b-a19d-854a2fdd2bf3
- Study
- Query Live Databases Using Plain English with AI
- Research task
- 86b9y6c99
- Tested at
- not recorded
- Source
- first-party
- Evidence state
- verified
- Proof shown
- input + output shown
- Cost / latency
- not captured
- Repeat run
- not captured
- Tester
- not captured
The last three rows are honest blanks, not placeholders — our capture has no field for them yet.
Query this
get_evidence({
tool: "basedash",
scenario: "live-database-plain-english-queries"
})MCP · mcp.aidemos.com/api/mcp
Free with attribution.
Same input, same check — 10 other tools
measured on Follow-Up Context
AI for Database✓ WorkedOn the next turn it restored the missing third customer correctly: the payment-methods answer includes Karan Joshi, and his $7,195.64 total matches the earlier ranking exactly.Anomaly AI✗ FailedIt did not keep the referent of 'the top 3' stable across turns: Follow-up 1 answered for Rahul Sharma, Priya Patel, and Deepak Kulkarni, but Follow-up 2 silently switched to Deepak Kulkarni, Rahul Sharma, and Karan Joshi.AskYourDatabase✓ WorkedIt preserves the selected top 3 across turns and reuses the earlier result set for the payment-method follow-up without issuing a new SQL query.BlazeSQL✓ WorkedIt retained the corrected top-3 customer set across both follow-ups, correctly using Deepak Kulkarni, Rahul Sharma, and Karan Joshi for the unpaid-order and payment-method lookups.camelAI✓ WorkedIt kept the follow-up scope correctly anchored to the top 3 highest-spending customers across both follow-ups, checking unpaid orders and payment methods only for Deepak Kulkarni, Rahul Sharma, and Karan Joshi.Definite✓ WorkedIt retained the top-3 customer context across both follow-ups, answering unpaid-order and payment-method questions for the same three customers from the initial list.Dot✓ WorkedIt carried the top 3 customers through both follow-ups and, on the payment-method question, correctly did not inherit the main query's paid/non-cancelled filter.Draxlr✓ WorkedIt kept the top 3 from that list context intact across both follow-ups and continued analyzing the same three customers.FutureSmart NL2SQL Agent✓ WorkedCarries the same top-3 customer trio forward into both follow-ups and reuses those same three customers in the later SQL filters.Querio✓ WorkedRetains follow-up context perfectly, reusing the same top-3 customer UUIDs in both later questions.
This evidence is published in
From the same study (page rebuilt from a later run)
Real inputs and real outputs, no retouching · every cell queryable via API & MCP · aidemos.com