Scenario definition · S44

The follow-up is ambiguous

A follow-up could refer to more than one plausible set from the previous turn.

What this scenario means

This scenario tests whether the agent notices ambiguous reference in conversation. When the previous reply produced more than one plausible set, a good agent does not guess which one is meant. It asks for clarification and waits for the user to choose a referent before continuing.

What we evaluate

  • Whether the agent recognises that the follow-up could refer to more than one prior set.
  • Whether it asks for clarification instead of selecting one referent implicitly.
  • Whether it avoids treating one candidate set as established before the ambiguity is resolved.
◌Test-case detail. Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

Capabilities this scenario exercises

A scenario may exercise one or more capabilities.

Capability

Conversational Interaction

Holds a working session rather than a series of isolated queries: context carried, questions refined, direction changed, ambiguity raised.

Benchmarks that use this scenario

A scenario has global identity and may be reused across benchmarks.

The follow-up is ambiguous — Scenario definition | AI Demos