Reported analytics match what actually happened
Whether reported analytics align with the underlying recorded activity.
What this scenario means
This scenario checks whether the product’s reported totals or analytics reconcile with what actually happened during the run. It is hard because a tool can answer ordinary questions correctly yet still miscount, omit events, or report numbers that do not match the record. A good product reports figures that match the observed activity and makes mismatches visible rather than hiding them.
What we evaluate
- Whether the reported figures match the activity that actually occurred.
- Whether the result reflects the same underlying events that were recorded.
- Whether discrepancies between reported numbers and recorded events are exposed rather than treated as correct.
Capabilities this scenario exercises
A scenario may exercise one or more capabilities.
Analytics & Observability
Lets whoever runs the agent see what people actually asked and what actually happened — and reports numbers that match reality. ON TRIAL: aggregate analytics are S57 and S24; individual inspection is S58 and S59. Demotes if the scenarios cannot discriminate. Deliberately NOT the same capability as the Customer Support benchmark's 'Analytics and observability' (C8) — there the operator is a different person from the user; here the buyer IS the user and sees every answer (rule C-10).
Analytics and observability
Lets the operator see what the agent did, inspect any conversation, and trust the reported numbers
Benchmarks that use this scenario
A scenario has global identity and may be reused across benchmarks.
AI Customer Support Chatbots
Automate customer support using an AI chatbot / AI customer support agent — which agents handle real customer support conversations for a business best?
AI Database Agents
Also uses this scenario.