Carries the same top-3 customer trio forward into both follow-ups and reuses those same three customers in the later SQL filters.
What was measured
Follow-Up Context
Does the tool remember previous answers correctly?
context, not decisivetransformation
Remembering prior turns improves workflow, but a tool can still do the core job without strong conversation memory. (2 of 3 judges)
What was given, what came back
Test input: Best customers with unpaid-order and payment-method follow-ups · text · group: live-database-plain-english-queries
Input — what we sent
The exact prompt
Who are my best customers — the ones who order the most and spend the most? Follow-up 1: For the top 3 from that list — do any of them have unpaid orders? Follow-up 2: What payment methods do these top 3 usually use?
A conversational multi-table customer analysis with follow-up questions. It asks for the best customers by both order volume and spend, then drills into unpaid orders for the top 3 and their usual payment methods. Designed to test ranking logic, join-heavy analysis, and follow-up context retention.
Why this input is hard
- · ambiguous business term interpretation
- · multi-table joins
- · aggregation and ranking
- · follow-up context retention
- · scoping to a selected subset
- · payment behavior analysis
Output — unretouched
Also checked on this input — same tool, 6 other criteria
Ambiguity Handling◐ MixedIt states a ranking heuristic in prose, using spend as the primary sort and order count as a secondary sort, but it still leaves the no-status-filter assumption unstated and does not ask the user to clarify the two-dimensional wording.Business Insight✓ WorkedExplains the ranking in business terms by calling out the high-value customer, the highest-order-count customer, and a loyalty/upsell targeting opportunity.Business Insight◐ MixedThe payment-method follow-up gives a concise dominant-method insight for the top customers, but the result is incomplete as a business answer because it does not surface the full payment-method picture implied by the customer totals.Plain English Query Handling✓ WorkedAccepts the best-customers question in plain English and returns a ranked answer without requiring SQL from the user.SQL Generation✓ WorkedGenerates a clean JOIN/GROUP BY/ORDER BY/LIMIT query for the top-customers ranking, showing the database-backed answer path is functioning on this turn.SQL Visibility✓ WorkedSurfaces the generated SQL in an Executed SQL panel with Copy Code, so the query can be inspected or copied directly.
Provenance
- Observation
- f1273e5e-f18a-444a-8beb-b3b8b303f544
- Evidence run
- af2abc96-3311-484b-a19d-854a2fdd2bf3
- Study
- Query Live Databases Using Plain English with AI
- Research task
- 86b9y6c99
- Tested at
- not recorded
- Source
- first-party
- Evidence state
- verified
- Proof shown
- input + output shown
- Cost / latency
- not captured
- Repeat run
- not captured
- Tester
- not captured
The last three rows are honest blanks, not placeholders — our capture has no field for them yet.
Query this
get_evidence({
tool: "futuresmart-nl2sql-agent",
scenario: "live-database-plain-english-queries"
})MCP · mcp.aidemos.com/api/mcp
Free with attribution.
Same input, same check — 10 other tools
measured on Follow-Up Context
AI for Database✗ FailedOn the first follow-up it dropped part of the top-3 context: the query result returned 4 rows for Deepak and Rahul only, yet the answer still asserted that Karan Joshi had no unpaid orders even though his row was absent.Anomaly AI✗ FailedIt did not keep the referent of 'the top 3' stable across turns: Follow-up 1 answered for Rahul Sharma, Priya Patel, and Deepak Kulkarni, but Follow-up 2 silently switched to Deepak Kulkarni, Rahul Sharma, and Karan Joshi.AskYourDatabase✓ WorkedIt preserves the selected top 3 across turns and reuses the earlier result set for the payment-method follow-up without issuing a new SQL query.Basedash◐ MixedIt remembered enough of the prior answer to check both ranking lists, but it still narrowed the follow-up instead of preserving the user's intended scope cleanly.BlazeSQL✓ WorkedIt retained the corrected top-3 customer set across both follow-ups, correctly using Deepak Kulkarni, Rahul Sharma, and Karan Joshi for the unpaid-order and payment-method lookups.camelAI✓ WorkedIt kept the follow-up scope correctly anchored to the top 3 highest-spending customers across both follow-ups, checking unpaid orders and payment methods only for Deepak Kulkarni, Rahul Sharma, and Karan Joshi.Definite✓ WorkedIt retained the top-3 customer context across both follow-ups, answering unpaid-order and payment-method questions for the same three customers from the initial list.Dot✓ WorkedIt carried the top 3 customers through both follow-ups and, on the payment-method question, correctly did not inherit the main query's paid/non-cancelled filter.Draxlr✓ WorkedIt kept the top 3 from that list context intact across both follow-ups and continued analyzing the same three customers.Querio✓ WorkedRetains follow-up context perfectly, reusing the same top-3 customer UUIDs in both later questions.
This evidence is published in
From the same study (page rebuilt from a later run)
Real inputs and real outputs, no retouching · every cell queryable via API & MCP · aidemos.com