The user supplies example questions and the queries they trust
A user supplies verified question-and-query examples that should shape later answers.
What this scenario means
This scenario tests whether the agent can learn from provided examples and apply that learning to later questions, not just replay the original wording. A good agent uses the trusted pair to improve future answers in the same topic and keeps that behaviour when the question is phrased differently.
What we evaluate
- Whether the agent uses the supplied example to improve a later answer on the same topic.
- Whether the agent handles a rephrased later question consistently with the supplied example.
- Whether the behaviour shows generalisation beyond the exact wording of the original example.
Capabilities this scenario exercises
A scenario may exercise one or more capabilities.
Training
Whether supplying company-specific information measurably improves future behaviour and generalises beyond the example given.
Benchmarks that use this scenario
A scenario has global identity and may be reused across benchmarks.
AI Database Agents
Which AI database agent answers business questions about a live relational database correctly — in the company's own terms, and honestly when the data cannot answer?