It successfully generated the ranking queries and follow-up lookups, including the top spenders, top order-count customers, unpaid-order status, and usual payment methods.
What was measured
SQL Generation
Does it generate database-backed SQL correctly?
decisive for this rankingtransformation
If the tool cannot produce correct SQL against the database, it cannot reliably answer the question being asked. (3 of 3 judges)
What was given, what came back
Test input: Best customers with unpaid-order and payment-method follow-ups · text · group: live-database-plain-english-queries
Input — what we sent
The exact prompt
Who are my best customers — the ones who order the most and spend the most? Follow-up 1: For the top 3 from that list — do any of them have unpaid orders? Follow-up 2: What payment methods do these top 3 usually use?
A conversational multi-table customer analysis with follow-up questions. It asks for the best customers by both order volume and spend, then drills into unpaid orders for the top 3 and their usual payment methods. Designed to test ranking logic, join-heavy analysis, and follow-up context retention.
Why this input is hard
- · ambiguous business term interpretation
- · multi-table joins
- · aggregation and ranking
- · follow-up context retention
- · scoping to a selected subset
- · payment behavior analysis
Output — unretouched
Also checked on this input — same tool, 5 other criteria
Ambiguity Handling⚠ StruggledIt did not ask what 'top 3 from that list' meant; instead it silently chose the spend ranking first and then added the order-count check.Business Insight✓ WorkedIt explains what the numbers mean by naming Rahul Sharma as the best overall customer, Mohan Vishe as the most frequent, and Deepak Kulkarni as the biggest spender.Follow-Up Context◐ MixedIt remembered enough of the prior answer to check both ranking lists, but it still narrowed the follow-up instead of preserving the user's intended scope cleanly.Plain English Query Handling✓ WorkedIt correctly handled a conversational, multi-part customer question without requiring SQL, producing two rankings plus follow-up answers.Result Readability✓ WorkedThe output stays readable for non-technical users, using compact tables and short summaries with explicit customer names and amounts.
Provenance
- Observation
- bff55c46-ffad-430e-a91c-0a9b7651909a
- Evidence run
- af2abc96-3311-484b-a19d-854a2fdd2bf3
- Study
- Query Live Databases Using Plain English with AI
- Research task
- 86b9y6c99
- Tested at
- not recorded
- Source
- first-party
- Evidence state
- verified
- Proof shown
- input + output shown
- Cost / latency
- not captured
- Repeat run
- not captured
- Tester
- not captured
The last three rows are honest blanks, not placeholders — our capture has no field for them yet.
Query this
get_evidence({
tool: "basedash",
scenario: "live-database-plain-english-queries"
})MCP · mcp.aidemos.com/api/mcp
Free with attribution.
Same input, same check — 6 other tools
measured on SQL Generation
AskYourDatabase✓ WorkedIt can generate follow-up SQL constrained to exactly the top 3 customers, using hardcoded customer IDs in the WHERE clause.BlazeSQL✓ WorkedIt generated a refined ranking query that joins customers to orders, filters out CANCELLED rows, requires PAID payment_status, and ranks customers by total_spent then total_orders.Dot✓ WorkedIt generated CTE-based SQL that joins customers to paid, non-cancelled orders, aggregates order_count and total_spend, and ranks customers across both measures.Draxlr✓ WorkedAcross the three turns it generated production-grade SQL, including ranking logic, unpaid-order checks, and payment-method aggregation with CTEs, ROW_NUMBER, COALESCE, and LEFT JOIN.FutureSmart NL2SQL Agent✓ WorkedGenerates a clean JOIN/GROUP BY/ORDER BY/LIMIT query for the top-customers ranking, showing the database-backed answer path is functioning on this turn.Querio✓ WorkedGenerates the ranking SQL and the follow-up SQL correctly across the three-turn customer analysis chain.
This evidence is published in
From the same study (page rebuilt from a later run)
Real inputs and real outputs, no retouching · every cell queryable via API & MCP · aidemos.com