Scenario definition · S43

The user changes direction mid-conversation

A conversation changes direction, and the agent must answer the new scope rather than carry the old one forward.

What this scenario means

This scenario tests whether the agent treats a conversation as stateful without keeping the wrong context. When the user redirects the question, the earlier subject or filter should be dropped if it no longer applies. A good agent answers for the new scope only, and does not silently blend the old scope into the new one.

What we evaluate

  • Whether the agent drops the earlier subject or filter when the user changes direction.
  • Whether the answer reflects only the new scope introduced by the follow-up.
  • Whether the agent avoids silently carrying forward the prior constraint into the new answer.
◌Test-case detail. Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

Capabilities this scenario exercises

A scenario may exercise one or more capabilities.

Capability

Conversational Interaction

Holds a working session rather than a series of isolated queries: context carried, questions refined, direction changed, ambiguity raised.

Benchmarks that use this scenario

A scenario has global identity and may be reused across benchmarks.

The user changes direction mid-conversation — Scenario definition | AI Demos