Scenario definition · S64

The connected data cannot answer the question

A user asks for information that the connected database does not hold.

What this scenario means

This scenario tests whether the agent recognises when a question is outside the database’s coverage. A good agent does not guess, fill in missing facts, or return a made-up value. It says the question cannot be answered from the available data.

What we evaluate

  • Whether the agent says the question cannot be answered from the available data.
  • Whether the agent avoids inventing a factual answer or number.
  • Whether the agent does not present an unrelated substitute as if it answered the question.
  • Whether the agent keeps the response tied to the missing data condition rather than pretending the database contains it.
◌Test-case detail. Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

Capabilities this scenario exercises

A scenario may exercise one or more capabilities.

Capability

Question Answering

Answers a question about the connected data correctly — finding the right data, using the company's meaning, matching the user's wording to what is stored — and says so when the data cannot answer.

Benchmarks that use this scenario

A scenario has global identity and may be reused across benchmarks.

The connected data cannot answer the question — Scenario definition | AI Demos