Scenario definition · S26
Correction changes future behavior where the product claims learning
A prior correction is expected to change how the agent answers later when the product claims to learn from feedback.
What this scenario means
This scenario checks whether feedback is only recorded, or whether it actually affects later behaviour. A good agent accepts the correction, carries it forward, and gives a later response that reflects the updated information instead of repeating the earlier mistake.
What we evaluate
- Whether the system accepts a correction after an earlier wrong answer.
- Whether a later interaction reflects the correction rather than the earlier error.
- Whether the earlier wrong behaviour is not simply repeated once feedback has been given.
- Whether the behaviour change appears in a later interaction when the product claims to learn from feedback.
◌Test-case detail. Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
Capabilities this scenario exercises
A scenario may exercise one or more capabilities.
Feedback and learning
Accepts corrections and improves future answers
Benchmarks that use this scenario
A scenario has global identity and may be reused across benchmarks.
AI Customer Support Chatbots
Automate customer support using an AI chatbot / AI customer support agent — which agents handle real customer support conversations for a business best?