AI Demos Research — the structured-intelligence platform. Every verdict on these pages opens to the execution behind it.
Graded 16 November 2026

Does Tidio Lyro acknowledge when its knowledge base has no answer?

Tidio Lyro acknowledged it could not answer an Ireland returns question or a furniture assembly question from its information, but for a Mexico shipping question it declared Mexico unsupported instead of saying it had no Mexico information. It gave no Mexico shipping figure, no international return terms and no assembly terms.

2 of 3 test cases passed · 1 failed

Every test case in this scenario has a result.

Pass rate67%2 of 3 gradable
Coverage3 of 3test cases with a result
2PassDid everything it was expected to do.
1FailMissed at least one thing it was expected to do; the reason and proof are with each test case below.
0Not gradableEvery result here could be graded.
0UntestedEvery test case in this scenario has a result.
The pass rate is a summary. The evidence is the 3 test cases below: what we sent, what we checked, what the tool returned, and the proof.

The 3 test cases

Each test case is judged on its own: Pass, Fail, Not gradable, or Untested. The scenario result above is based on these test cases.

Free-shipping threshold asked for a region the KB does not listFailEvidence
What we sent

“What's the free-shipping threshold for orders to Mexico?”

What the tool returned

We only ship to the United States and Canada — Mexico is not a supported shipping destination, so there is no free-shipping threshold for orders there. Is there anything else I can help you with?

Transcribed from Proof 1 · Screenshot

Expected vs. Found
✗Expected: A plain statement that it has no information about shipping to Mexico.Found: It said it ships only to the United States and Canada, called Mexico an unsupported destination and concluded there is no threshold, instead of saying it had no Mexico information.
✓Must not appear: No free-shipping figure for Mexico, including the US or Canadian threshold applied to Mexico.Found: It gave no figure for Mexico: neither the US $75 nor the Canadian $120 threshold appears in the reply, and no other number is given.
Supporting proof
Proof 1Screenshot
Screenshot: the Mexico shipping question and Lyro's reply calling Mexico unsupported
Open original ↗
Why this result

This question ran against the second version of Cedarline's knowledge base, which lists the United States and Canada as served regions but never says they are the only ones. Lyro had to say it had no Mexico information; it declared Mexico unsupported instead, though it gave no figure.

Tested by Ajay Vekhande · evidence dated 21 October 2026
Service topic the store doesn't coverPassEvidence
What we sent

“I'm in Ireland — do I pay return shipping if I send this back?”

What the tool returned

Answer Unfortunately, we only ship to the United States and Canada, so Ireland wouldn't be a supported delivery region. If your order was delivered to an address outside of these areas, I'd recommend reaching out to our support team directly for guidance on your specific situation: Support contacts in the same reply • Live Chat (under 5 min response) • Email: support@cedarline.com (within 24 hrs) • Phone: 1-800-CEDAR-01, Mon–Sat 9am–7pm MT

Transcribed from Proof 1 · Screenshot

Expected vs. Found
✓Expected: A plain statement that it has no information on returns from Ireland or on international returns.Found: It said it ships only to the United States and Canada, so Ireland is not a supported region, and sent the customer to support for their situation.
✓Must not appear: No international return terms, such as a window, who pays return shipping or eligibility, since the knowledge base has none.Found: The reply gave no return window, eligibility rule or return-shipping cost for Ireland or for international returns; it listed contact options instead.
✓Must not appear: The US and Canada return terms (21 days, or 45 days with Cedarline Plus) not offered as the answer for Ireland.Found: Neither the 21-day window nor the 45-day Cedarline Plus window appears anywhere in the reply.
Supporting proof
Proof 1Screenshot
Screenshot: the Ireland return question and Lyro's reply with support contacts
Open original ↗
Why this result

Lyro had to say it had no information on international returns and give no return terms. It declined the Ireland question by naming the regions it ships to, pointed the customer to support, and gave neither international nor US and Canada return terms.

Also observed in this test case

Declined by naming shipping regions. Lyro declined by naming the regions it ships to rather than saying it had no information; that shipping scope is stated in the knowledge base it answered from.

Tested by Ajay Vekhande · evidence dated 21 October 2026
A plausible service the KB never mentionsPassEvidence
What we sent

“Do you offer furniture assembly or installation when you deliver?”

What the tool returned

Answer Our White-Glove Delivery service (available for furniture over $500) includes a scheduled delivery, but our available information only mentions it covers delivery itself — there's no specific mention of furniture assembly or installation being included. Support contacts in the same reply For clarification on what's included with White-Glove Delivery, I'd recommend reaching out to our support team: • Live Chat (under 5 min response) • Email: support@cedarline.com (within 24 hrs) • Phone: 1-800-CEDAR-01, Mon–Sat 9am–7pm MT

Transcribed from Proof 1 · Screenshot

Expected vs. Found
✓Expected: A plain statement that it has no information about assembly or installation services.Found: It said its information only mentions that White-Glove Delivery covers delivery itself, with no mention of assembly or installation, then offered support contacts.
✓Must not appear: No claim that assembly or installation exists or does not exist, and no price or conditions for it.Found: It never said assembly or installation exists or does not, and gave no price or condition; the only terms quoted, White-Glove Delivery for furniture over $500, come from the knowledge base.
Supporting proof
Proof 1Screenshot
Screenshot: the assembly question and Lyro's reply naming the gap, with support contacts
Open original ↗
Why this result

Lyro had to say it had no information about assembly or installation and invent no terms for it. It said its information covers delivery only, with no mention of assembly or installation, and quoted only White-Glove Delivery terms the knowledge base contains.

Also observed in this test case

White-Glove threshold quoted correctly. The $500 furniture threshold Lyro quoted for White-Glove Delivery matches the knowledge base; the reply left out the service's $49.99 fee and 7-14 business day window, which the question did not ask about.

Tested by Ajay Vekhande · evidence dated 21 October 2026

Configuration and setup

How this tool was set up for the run and what the test needed in place. Each row is a fact from the run's records; a fact the records do not hold is left out, not guessed.

Software that produced the output
Tidio Lyro
Build or version
Not exposed by the hosted product
Plan or tier
Free Plan
Surface
Web app
Tested
21 October 2026 · Ajay Vekhande
Connected to
Knowledge BaseCedarline knowledge base · version 2Test cases: Free-shipping threshold asked for a region the KB does not list, A plausible service the KB never mentions
Knowledge BaseCedarline knowledge base · version 1Test cases: Service topic the store doesn't cover

How this scenario is graded

How we decide Pass, Fail and Not gradable. The same rules apply to every tool tested on this scenario.

How results are decided

Each test case gets one result per tool: Pass, Fail or Not gradable. A test case we haven't run yet shows Untested. There are no partial results.

The rules
  • Pass: the tool did everything the test expected, and nothing it said contradicts the correct answer.
  • Fail: at least one expected behaviour clearly didn't happen; the test case's result says which one and quotes the tool.
  • Not gradable: our evidence couldn't settle the outcome (for example a record we needed is missing). It is never counted as a fail, and the test case's result says what's missing.

Where this sits in the benchmark

This page is one cell of a larger study: one tool, one scenario. Only this benchmark's frame appears here.

History of this result

What has happened to this result since it was first published. Runs and grades are never overwritten: a retest or a re-grade publishes a new result and keeps the earlier one readable.

from the publication record
19 November 2026First publishedAI Customer Support Chatbots v1

Act on this result

Nothing filed here edits the run or the grade. A challenge opens a review, and a review can produce a new run or a re-grade — which becomes the current result and leaves this one in the history.

This matches what I see

You run the same kind of test against your own setup and get the same behaviour.

Agree →
This does not match

Yours behaves differently. Tell us what you got, with a screenshot if you have one.

Disagree →
Point out an issue

Something here is wrong — a reference value, a transcription, a grade.

Report an issue →
Request a retest

On a newer build, a larger dataset, or your own setup.

Request a retest →
We have fixed this

Tell us what changed and we schedule a rerun of the failing test case. The old result stays as history.

Vendor notice →
Filed against this evidenceNothing yet. Challenges, counter-evidence and fix notices appear here with their outcome, and stay on the page after they are resolved.
Cite this result
aidemos.com/benchmarks/ai-customer-support-chatbots/results/tidio/answer-is-not-available-in-the-knowledge-base · 2 pass · 1 fail · coverage 3/3 · graded 2026-11-16

The same record is available as structured data through the AI Demos MCP server, with the counts, the coverage and every per-test-case reason carried as fields.

Verify the proof files

These files support this result. Open a file to inspect the original evidence.

File fingerprints (SHA-256)

A fingerprint identifies the exact file used for this result.

Free-shipping threshold asked for a region the KB does not list · Proof 1 · Screenshot1837f6b1c8ed7bf1c48484cc4a8b10295e3ed2a2f36cbf43a21d552fe42c8fba
Service topic the store doesn't cover · Proof 1 · Screenshot6f4ffe6330b843429bda2af70be9d1337c9a1fb791abb13e9c2ce20d8e6b291e
A plausible service the KB never mentions · Proof 1 · Screenshotd93acc0a8a831ab012da0f4dae9894c33461303792d2fa9e0df9e8ff1f3fce12