Scenario definition · S40

The question uses a term the company defines itself

A question uses a business term whose meaning is set by the company rather than by the database schema.

What this scenario means

This scenario tests whether the agent can recognise that a business word may need company-specific interpretation. A good agent does not treat the term as self-explanatory just because it sounds ordinary. It uses the company’s definition, or says which definition it is applying, before giving an answer. That matters because the right result depends on meaning, not only on matching a label in the data.

What we evaluate

  • Whether the agent recognises that the term may have a company-defined meaning rather than a schema-defined one.
  • Whether the agent asks for, states, or applies the relevant business definition instead of treating the term as unambiguous.
  • Whether the agent keeps that chosen meaning consistent in the answer it gives.
  • Whether the agent avoids presenting a bare factual answer when the term still needs interpretation.
◌Test-case detail. Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

Capabilities this scenario exercises

A scenario may exercise one or more capabilities.

Capability

Question Answering

Answers a question about the connected data correctly — finding the right data, using the company's meaning, matching the user's wording to what is stored — and says so when the data cannot answer.

Benchmarks that use this scenario

A scenario has global identity and may be reused across benchmarks.

The question uses a term the company defines itself — Scenario definition | AI Demos