Tool in benchmark · Version 1

Tidio Lyro in AI Customer Support Chatbots

Scenario-level performance from current published Results.

8Passed
0Failed
2Different test set
16No published Result yet

10 of 26 scenarios published

CapabilityScenariosResult
Agent configuration and control2 pass · 0 fail · 2 of 2 published
Analytics and observability1 pass · 0 fail · 1 of 3 published
Escalation and human handoff2 pass · 0 fail · 2 of 3 published
Intent understanding3 pass · 0 fail · 3 of 3 published
Knowledge-grounded answering0 pass · 0 fail · 2 of 3 published
Action execution0 of 5 published
Conversation continuity0 of 3 published
Feedback and learning0 of 2 published
Multilingual support0 of 2 published
PassedFailedPublished, different test setNo published Result yet

Tidio Lyro passed every graded scenario it has a published Result for.

How Tidio Lyro performed

Open a capability to explore its scenarios. Each row reports the test set in its published Result.

Agent configuration and control2 pass
ScenarioPublished outcomesTest coverageResult
Configured instruction changes agent behavior
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Tidio Lyro followed a custom instruction when answering a return request. It refused an opened-candle return without suggesting that the item could still be returned.

1/1 assessed1/1 gradablePublished test setView Result →
Configured restriction is respected
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Tidio Lyro refused the delivery-address change without performing or offering any part of it. It explained that address changes were unavailable for any order, regardless of status.

1/1 assessed1/1 gradablePublished test setView Result →
Analytics and observability1 pass2 no Result yet
ScenarioPublished outcomesTest coverageResult
Conversation activity can be inspected
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Tidio Lyro let an operator inspect an earlier customer conversation from start to finish. The shipped-order chat appeared in its conversation log, with order details and the return policy available in the transcript.

1/1 assessed1/1 gradablePublished test setView Result →
Agent actions/tool calls can be inspectedNo published result
Reported analytics match what actually happenedNo published result
Escalation and human handoff2 pass1 no Result yet
ScenarioPublished outcomesTest coverageResult
Agent cannot resolve the request
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Tidio Lyro acknowledged it could not resolve the account email change and directed the customer to human help. It supplied support options without inventing self-service steps or claiming to have made the change.

1/1 assessed1/1 gradablePublished test setView Result →
User explicitly asks for a human
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Tidio Lyro passed the human-handoff scenario. The customer asked for a human, Lyro answered once with a direct transfer, and the same conversation then reached a human agent in Tidio's inbox.

1/1 assessed1/1 gradablePublished test setView Result →
A configured rule requires escalationNo published result
Intent understanding3 pass
ScenarioPublished outcomesTest coverageResult
One message contains multiple requests
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Tidio Lyro handled the one tested multi-request message successfully. It answered the order status and return policy in one reply, and the order details matched the live order data while the policy block covered the requested rules.

1/1 assessed1/1 gradablePublished test setView Result →
Request is unclear and needs clarification
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Tidio Lyro sought clarification rather than acting on an unclear cancellation request. It offered the customer a choice of an order, a membership or a monthly product subscription.

1/1 assessed1/1 gradablePublished test setView Result →
Request requires choosing the correct source or action
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Tidio Lyro used the order system, not the knowledge base, and correctly reported the order as shipped. Its answer included tracking information and an estimated delivery date.

1/1 assessed1/1 gradablePublished test setView Result →
Knowledge-grounded answering2 different test set1 no Result yet
ScenarioPublished outcomesTest coverageResult
Answer is available in the knowledge baseDifferent test setPublished Result available, but not included in this comparison.View Result →
Answer is not available in the knowledge baseDifferent test setPublished Result available, but not included in this comparison.View Result →
The request uses different wording than the sourceNo published result
Action execution5 no Result yet
ScenarioPublished outcomesTest coverageResult
Action requires confirmationNo published result
Requested action cannot be completedNo published result
Retrieve information from an external systemNo published result
Update something in an external systemNo published result
User is not authorized to perform the actionNo published result
Conversation continuity3 no Result yet
ScenarioPublished outcomesTest coverageResult
Customer corrects information given earlierNo published result
Earlier conversation/session needs to be continuedNo published result
Information from earlier in the conversation is needed laterNo published result
Feedback and learning2 no Result yet
ScenarioPublished outcomesTest coverageResult
Correction changes future behavior where the product claims learningNo published result
Feedback can be submittedNo published result
Multilingual support2 no Result yet
ScenarioPublished outcomesTest coverageResult
Customer communicates in another supported languageNo published result
Customer switches languages during the conversationNo published result

Reading these Results

Published evidence and test coverage answer different questions.

Publication availability

Which scenarios have a Result?

A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.

Test coverage

What does each Result cover?

Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.

Scenario scope

Out of 26 scenarios

The summary counts each of the 26 scenarios once. A scenario fails when any of its published test cases failed. “No published Result yet” means nothing is public for that scenario; it does not say whether Tidio Lyro was tested on it.

Tidio Lyro in AI Customer Support Chatbots | AI Demos