Kept the top-3 customer context across the follow-up chain by hardcoding the same three customer IDs into the unpaid-order check.
What was measured
Follow-Up Context
Does the tool remember previous answers correctly?
transformation
What was given, what came back
Test input: Best customers with unpaid-order and payment-method follow-ups · text · group: ecommerce-nl2sql-benchmark
Input — what we sent
The exact prompt
Who are my best customers — the ones who order the most and spend the most? Follow-up 1: For the top 3 from that list — do any of them have unpaid orders? Follow-up 2: What payment methods do these top 3 usually use?
A conversational multi-table customer analysis that identifies best customers by both order volume and spend, then drills into unpaid orders and payment methods for the top 3.
Why this input is hard
- · Ambiguous business-term interpretation
- · Multi-table joins across customers orders and payments
- · Aggregation and ranking
- · Follow-up context retention
- · Scoped drill-down to the top 3 customers
- · Readable customer-level output
Output — unretouched
Also checked on this input — same tool, 9 other criteria
Ambiguity Handling✓ WorkedHandled the ambiguous phrase by not collapsing it into one ranking; it produced separate order-count and total-spend rankings instead of guessing silently.Business Insight✓ WorkedGenerated useful takeaways automatically, including Rahul Sharma as the standout all-rounder, Mohan Vishe as the most loyal frequent buyer, and Vikram Singh as the largest spender.Business Insight✓ WorkedThe follow-up reasoning went beyond listing payments by flagging Mohan Vishe's unpaid shipped orders as a process risk and noting that Rahul Sharma's pattern was largely UPI-driven.Chart / Visualization Support✗ FailedVisualization did not auto-generate on this customer analysis flow; the report says it required an additional prompt every time.Plain English Query Handling✓ WorkedThe tool understood follow-up phrasing naturally, including 'For the top 3 from that list' and 'What payment methods do these top 3 usually use?'.Plain English Query Handling✓ WorkedAccepted an informal customer-analytics question and split it into two dimensions without requiring SQL: who orders most and who spends most.Result Readability✓ WorkedPresented the follow-up results as clearly labeled customer status blocks such as 'All Clear', '1 Unpaid', and 'All 4 Unpaid!', which makes the risk scan easy.Result Readability✓ WorkedThe customer ranking output was easy to scan because it separated frequent shoppers from highest spenders into clearly named tables with totals and ranks.SQL Generation✓ WorkedRan 2 SQL queries simultaneously for the main request, and then generated a filtered follow-up query for the exact top 3 customers.
Provenance
- Observation
- 0f379e1b-c980-49c7-9d60-df438430e631
- Evidence run
- db2bb5d5-0e0e-4cb3-8d76-3555c45c23cd
- Study
- Query Live Databases Using Plain English with AI
- Research task
- 86b9y6c99
- Tested at
- not recorded
- Source
- first-party
- Evidence state
- verified
- Proof shown
- input + output shown
- Cost / latency
- not captured
- Repeat run
- not captured
- Tester
- not captured
The last three rows are honest blanks, not placeholders — our capture has no field for them yet.
Query this
get_evidence({
tool: "askyourdatabase",
scenario: "ecommerce-nl2sql-benchmark"
})MCP · mcp.aidemos.com/api/mcp
Free with attribution.
Same input, same check — 3 other tools
measured on Follow-Up Context
Basedash◐ MixedIt retained the earlier ranking context only partially: the unpaid-orders follow-up checked the top 3 highest spenders first, then had to run a separate pass for the top 3 by order count instead of carrying one unambiguous 'top 3' thread forward.Definite✓ WorkedIt retained context across follow-ups, allowing the user to ask about unpaid orders and then payment methods for the top 3 from the earlier spend-ranked list.Querio✓ WorkedCarries the top-3 customer identities forward across follow-ups and regenerates fresh SQL on each turn, preserving context correctly across at least two drill-down questions.
This evidence is published in
From the same study (page rebuilt from a later run)
Real inputs and real outputs, no retouching · every cell queryable via API & MCP · aidemos.com