Does Intercom Fin say when the answer isn't in the knowledge base?
Intercom Fin passed all three checks. It said the answer was unavailable in each case and did not invent one: Mexico had no free-shipping figure, Ireland had no return-shipping terms, and assembly or installation was not included with delivery.
3 of 3 test cases passed
Every test case in this scenario has a result.
The 3 test cases
Each test case is judged on its own: Pass, Fail, Not gradable, or Untested. The scenario result above is based on these test cases.
Free-shipping threshold asked for a region the KB does not listPassEvidence
“What's the free-shipping threshold for orders to Mexico?”
The tool's reply is shown in the evidence under Proof.
Free standard shipping isn't available for orders to Mexico right now. Our current free-shipping thresholds are: United States: orders over $75 Canada: orders over $120 Cedarline currently ships to delivery addresses in the contiguous United States and Canada only, so Mexico isn't included in the free-shipping regions. Is that what you were looking for?
Transcribed from Proof 1 · Screenshot
| ✓ | Expected: an explicit statement that it has no information on shipping to MexicoFound: It said free standard shipping is unavailable for Mexico and that only the contiguous United States and Canada are covered. |
| ✓ | Expected: any figure given for Mexico, including the US or Canadian threshold applied to MexicoFound: It gave only the United States and Canada figures and did not apply either one to Mexico. |
It needed to say there was no Mexico threshold and avoid inventing one. The reply says free shipping is unavailable for Mexico, lists only US and Canada thresholds, and leaves Mexico without a figure.
Regional figures. It listed separate United States and Canada figures, so neither figure was presented as Mexico's.
Service topic the store doesn't coverPassEvidence
“I'm in Ireland — do I pay return shipping if I send this back?”
The tool's reply is shown in the evidence under Proof.
Our return policy covers the return window, restocking fees, and refund timing, but it does not state who pays return shipping for international returns like Ireland. We do ship to the United States and Canada, and our shipping info is based on those regions. If the item is damaged, defective, or wrong, you can report it within 7 days with photos, and no restocking fee applies. Is the return because you changed your mind, or because the item arrived damaged, defective, or wrong?
Transcribed from Proof 1 · Screenshot
| ✓ | Expected: an explicit statement that it has no information on returns from IrelandFound: It said the policy does not say who pays return shipping for Ireland. |
| ✓ | Expected: any international returns termFound: It gave no Ireland-specific return window, payer rule, or eligibility condition. |
| ✓ | Expected: the US/CA return terms offered as the answer to the Ireland questionFound: It did not use the US and Canada return window as the answer to the Ireland question. |
It needed to say the content had no Ireland return-shipping answer and not substitute domestic terms. The reply says it does not state who pays shipping for Ireland, gives no Ireland-specific return terms, and does not offer the US/Canada window as the answer.
General terms only. It also gave general damage-report conditions without an Ireland-specific qualifier, so they could be read as applying.
A plausible service the KB never mentionsPassEvidence
“Do you offer furniture assembly or installation when you deliver?”
The tool's reply is shown in the evidence under Proof.
We do offer a White-Glove Delivery option for furniture orders over $500. It costs $49.99 and includes scheduled delivery within a 7–14 business day window. The description only confirms scheduled delivery timing for this service. It doesn't mention furniture assembly or installation being included with delivery. So if you choose White-Glove Delivery, you can expect a scheduled delivery appointment, but assembly isn't listed as part of the service.
Transcribed from Proof 1 · Screenshot
| ✓ | Expected: an explicit statement that it has no information about assembly or installation servicesFound: It said the service description does not mention assembly or installation and that assembly is not listed. |
| ✓ | Expected: any statement that the service exists or does not exist, its price or conditionsFound: It gave White-Glove Delivery's price and timing, but kept assembly out of the service. |
It needed to say assembly or installation was not covered and not invent service details. The reply says the service description does not mention assembly, gives only White-Glove Delivery's own price and timing, and keeps assembly outside that service.
Near negative claim. It said assembly was not listed as part of the service, framing the point as what the listing covers.
Configuration and setup
How this tool was set up for the run and what the test needed in place. Each row is a fact from the run's records; a fact the records do not hold is left out, not guessed.
How this scenario is graded
How we decide Pass, Fail and Not gradable. The same rules apply to every tool tested on this scenario.
Each test case gets one result per tool: Pass, Fail or Not gradable. A test case we haven't run yet shows Untested. There are no partial results.
- Pass: the tool did everything the test expected, and nothing it said contradicts the correct answer.
- Fail: at least one expected behaviour clearly didn't happen; the test case's result says which one and quotes the tool.
- Not gradable: our evidence couldn't settle the outcome (for example a record we needed is missing). It is never counted as a fail, and the test case's result says what's missing.
Where this sits in the benchmark
This page is one cell of a larger study: one tool, one scenario. Only this benchmark's frame appears here.
| Level | Name | Scope |
|---|---|---|
| Benchmark | AI Customer Support Chatbots → | 26 scenarios · 13 tools |
| Capability | Knowledge-grounded answering → | |
| Scenario | Answer is not available in the knowledge base → | |
| Tool | Intercom Fin → |
Global scenario definition → · Global capability definition → · Intercom Fin product page →
History of this result
What has happened to this result since it was first published. Runs and grades are never overwritten: a retest or a re-grade publishes a new result and keeps the earlier one readable.
Act on this result
Nothing filed here edits the run or the grade. A challenge opens a review, and a review can produce a new run or a re-grade — which becomes the current result and leaves this one in the history.
You run the same kind of test against your own setup and get the same behaviour.
Agree →Yours behaves differently. Tell us what you got, with a screenshot if you have one.
Disagree →Something here is wrong — a reference value, a transcription, a grade.
Report an issue →Tell us what changed and we schedule a rerun of the failing test case. The old result stays as history.
Vendor notice →The same record is available as structured data through the AI Demos MCP server, with the counts, the coverage and every per-test-case reason carried as fields.
Verify the proof files
These files support this result. Open a file to inspect the original evidence.
File fingerprints (SHA-256)
A fingerprint identifies the exact file used for this result.