Tool in benchmark · Version 1

Intercom Fin in AI Customer Support Chatbots

Scenario-level performance from current published Results.

17 scenarios with published Results · 26 scenarios in the benchmark

How Intercom Fin performed

Open a capability to explore its scenarios. Each row reports the test set in its published Result; counts are not combined into an overall score.

Action execution5 scenarios · 3 with published Results
ScenarioPublished outcomesTest coverageResult
Action requires confirmation
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin required confirmation before carrying out the cancellation. It asked before acting, cancelled only after the customer agreed, and the subscription record shows the status changed to cancelling.

1/1 assessed1/1 gradablePublished test setView Result →
Requested action cannot be completed
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin refused the address change instead of claiming it worked. It said the order had already shipped, gave next steps, and the order state shows shipped with address changes disabled.

1/1 assessed1/1 gradablePublished test setView Result →
Retrieve information from an external system
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin retrieved the shipped order's live status from the external system. The reply and the logged 200 call match on shipped status, UPS, tracking number, shipped date, and estimated delivery, showing the answer came from that call.

1/1 assessed1/1 gradablePublished test setView Result →
Update something in an external systemNo published result
User is not authorized to perform the actionNo published result
Agent configuration and control2 scenarios · 2 with published Results
ScenarioPublished outcomesTest coverageResult
Configured instruction changes agent behavior
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin followed the configured instruction in this case. It told the customer opened candles were final sale and did not offer a refund, matching the live guidance for home decor items on all channels.

1/1 assessed1/1 gradablePublished test setView Result →
Configured restriction is respected
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin respected the configured restriction in this case. It declined the delivery-address change, said such changes were not supported after checkout, and offered help with other order or shipping questions instead.

1/1 assessed1/1 gradablePublished test setView Result →
Analytics and observability3 scenarios · 2 with published Results
ScenarioPublished outcomesTest coverageResult
Agent actions/tool calls can be inspected
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin shows the agent’s tool call in this scenario. The order lookup is linked to the open conversation and returned successfully.

1/1 assessed1/1 gradablePublished test setView Result →
Reported analytics match what actually happened
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin’s analytics matched the recorded conversation and escalation counts for this period. Its conversation report showed 46 of 46 items, and its escalation report showed 2 of 2, with the comparison supported by the visible query document.

1/1 assessed1/1 gradablePublished test setView Result →
Conversation activity can be inspectedNo published result
Conversation continuity3 scenarios · 3 with published Results
ScenarioPublished outcomesTest coverageResult
Customer corrects information given earlier
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin handled the corrected order lookup correctly. After the customer corrected the number, it answered only about order 41830 with processing status and no carrier or tracking number, and left out order 41982's shipped details.

1/1 assessed1/1 gradablePublished test setView Result →
Earlier conversation/session needs to be continued
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin continued the earlier return conversation after the widget was closed and reopened. The follow-up reply stayed on return timing and did not ask again for the order number or item.

1/1 assessed1/1 gradablePublished test setView Result →
Information from earlier in the conversation is needed later
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin passed this scenario. It answered the later question using the earlier order number and gave the shipping status, delivery date, and tracking number without asking for the number again.

1/1 assessed1/1 gradablePublished test setView Result →
Escalation and human handoff3 scenarios · 1 with published Results
ScenarioPublished outcomesTest coverageResult
User explicitly asks for a human
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin handled a direct request for a human. It offered a human handoff on the first reply, showed a one-tap connect option, and the same conversation later appeared in the agent inbox with a teammate reply.

1/1 assessed1/1 gradablePublished test setView Result →
A configured rule requires escalationNo published result
Agent cannot resolve the requestNo published result
Feedback and learning2 scenarios · 2 with published Results
ScenarioPublished outcomesTest coverageResult
Correction changes future behavior where the product claims learning
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin changed the answer in a new customer conversation after the correction was saved. The later chat gave the corrected 30-day return window and 45 days for Cedarline Plus, while the earlier answer had said 21 days from the same private sources.

1/1 assessed1/1 gradablePublished test setView Result →
Feedback can be submitted
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin lets a customer submit feedback on a wrong answer. It opened the answer-improvement panel, let a correction be entered as a new snippet in English, and showed the saved 30-day version; another saved correction was also visible.

1/1 assessed1/1 gradablePublished test setView Result →
Intent understanding3 scenarios · 1 with published Results
ScenarioPublished outcomesTest coverageResult
One message contains multiple requests
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin answered both requests in one message. It gave the shipping status and return policy in one reply, and kept the two parts separately legible.

1/1 assessed1/1 gradablePublished test setView Result →
Request is unclear and needs clarificationNo published result
Request requires choosing the correct source or actionNo published result
Knowledge-grounded answering3 scenarios · 1 with published Results
ScenarioPublished outcomesTest coverageResult
Answer is not available in the knowledge baseDifferent test setPublished Result available, but not included in this comparison.View Result →
Answer is available in the knowledge baseNo published result
The request uses different wording than the sourceNo published result
Multilingual support2 scenarios · 2 with published Results
ScenarioPublished outcomesTest coverageResult
Customer communicates in another supported language
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin answered the French support question in French. The reply matched the English KB on the return window, eligible condition, restocking fee, non-returnable items, refund timing, and damaged-item path, and it stayed French through the closing line.

1/1 assessed1/1 gradablePublished test setView Result →
Customer switches languages during the conversation
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

Intercom Fin fails to keep replying in the customer's current language when the customer switches languages. It kept the return-policy context, but answered the English follow-up in French in both the default conversation and the retest with the language-match guidance.

1/1 assessed1/1 gradablePublished test setView Result →

Reading these Results

Published evidence and test coverage answer different questions.

Publication availability

Which scenarios have a Result?

A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.

Test coverage

What does each Result cover?

Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.

Scenario scope

Inventory is not testing progress

The 26 scenarios describe this benchmark’s scope. They are not an assumed applicability or test-coverage denominator for Intercom Fin.

Intercom Fin in AI Customer Support Chatbots | AI Demos