Intercom Fin in AI Customer Support Chatbots
Scenario-level performance from current published Results.
17 scenarios with published Results · 26 scenarios in the benchmark
How Intercom Fin performed
Open a capability to explore its scenarios. Each row reports the test set in its published Result; counts are not combined into an overall score.
Action execution5 scenarios · 3 with published Results
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Action requires confirmation | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedIntercom Fin required confirmation before carrying out the cancellation. It asked before acting, cancelled only after the customer agreed, and the subscription record shows the status changed to cancelling. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Requested action cannot be completed | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedIntercom Fin refused the address change instead of claiming it worked. It said the order had already shipped, gave next steps, and the order state shows shipped with address changes disabled. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Retrieve information from an external system | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedIntercom Fin retrieved the shipped order's live status from the external system. The reply and the logged 200 call match on shipped status, UPS, tracking number, shipped date, and estimated delivery, showing the answer came from that call. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Update something in an external system | No published result | ||
| User is not authorized to perform the action | No published result | ||
Agent configuration and control2 scenarios · 2 with published Results
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Configured instruction changes agent behavior | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedIntercom Fin followed the configured instruction in this case. It told the customer opened candles were final sale and did not offer a refund, matching the live guidance for home decor items on all channels. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Configured restriction is respected | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedIntercom Fin respected the configured restriction in this case. It declined the delivery-address change, said such changes were not supported after checkout, and offered help with other order or shipping questions instead. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Analytics and observability3 scenarios · 2 with published Results
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Agent actions/tool calls can be inspected | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedIntercom Fin shows the agent’s tool call in this scenario. The order lookup is linked to the open conversation and returned successfully. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Reported analytics match what actually happened | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedIntercom Fin’s analytics matched the recorded conversation and escalation counts for this period. Its conversation report showed 46 of 46 items, and its escalation report showed 2 of 2, with the comparison supported by the visible query document. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Conversation activity can be inspected | No published result | ||
Conversation continuity3 scenarios · 3 with published Results
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Customer corrects information given earlier | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedIntercom Fin handled the corrected order lookup correctly. After the customer corrected the number, it answered only about order 41830 with processing status and no carrier or tracking number, and left out order 41982's shipped details. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Earlier conversation/session needs to be continued | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedIntercom Fin continued the earlier return conversation after the widget was closed and reopened. The follow-up reply stayed on return timing and did not ask again for the order number or item. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Information from earlier in the conversation is needed later | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedIntercom Fin passed this scenario. It answered the later question using the earlier order number and gave the shipping status, delivery date, and tracking number without asking for the number again. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Escalation and human handoff3 scenarios · 1 with published Results
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| User explicitly asks for a human | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedIntercom Fin handled a direct request for a human. It offered a human handoff on the first reply, showed a one-tap connect option, and the same conversation later appeared in the agent inbox with a teammate reply. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| A configured rule requires escalation | No published result | ||
| Agent cannot resolve the request | No published result | ||
Feedback and learning2 scenarios · 2 with published Results
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Correction changes future behavior where the product claims learning | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedIntercom Fin changed the answer in a new customer conversation after the correction was saved. The later chat gave the corrected 30-day return window and 45 days for Cedarline Plus, while the earlier answer had said 21 days from the same private sources. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Feedback can be submitted | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedIntercom Fin lets a customer submit feedback on a wrong answer. It opened the answer-improvement panel, let a correction be entered as a new snippet in English, and showed the saved 30-day version; another saved correction was also visible. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Intent understanding3 scenarios · 1 with published Results
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| One message contains multiple requests | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedIntercom Fin answered both requests in one message. It gave the shipping status and return policy in one reply, and kept the two parts separately legible. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Request is unclear and needs clarification | No published result | ||
| Request requires choosing the correct source or action | No published result | ||
Knowledge-grounded answering3 scenarios · 1 with published Results
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Answer is not available in the knowledge base | Different test setPublished Result available, but not included in this comparison. | View Result → | |
| Answer is available in the knowledge base | No published result | ||
| The request uses different wording than the source | No published result | ||
Multilingual support2 scenarios · 2 with published Results
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Customer communicates in another supported language | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedIntercom Fin answered the French support question in French. The reply matched the English KB on the return window, eligible condition, restocking fee, non-returnable items, refund timing, and damaged-item path, and it stayed French through the closing line. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Customer switches languages during the conversation | 0 Pass1 Fail0 Not gradable 1 of 1 test case failedWhat happenedIntercom Fin fails to keep replying in the customer's current language when the customer switches languages. It kept the return-policy context, but answered the English follow-up in French in both the default conversation and the retest with the language-match guidance. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Reading these Results
Published evidence and test coverage answer different questions.
Which scenarios have a Result?
A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.
What does each Result cover?
Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.
Inventory is not testing progress
The 26 scenarios describe this benchmark’s scope. They are not an assumed applicability or test-coverage denominator for Intercom Fin.