Scenario in benchmark · AI Customer Support Chatbots · Version 1
Correction changes future behavior where the product claims learning
A prior correction is expected to change how the agent answers later when the product claims to learn from feedback.
What we check
- Whether the system accepts a correction after an earlier wrong answer.
- Whether a later interaction reflects the correction rather than the earlier error.
- Whether the earlier wrong behaviour is not simply repeated once feedback has been given.
- Whether the behaviour change appears in a later interaction when the product claims to learn from feedback.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
Freshdesk FreddyFailed1 of 1 test case failedOffered no way to correct a past answer, so the correction-and-follow-up scenario could not be carried out.View Result →
No follow-up question was sent in a new conversation.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
KommunicateFailed1 of 1 test case failedFailed to carry a correction into a later answer.View Result →
Its feedback area only showed thumbs up or thumbs down on the earlier answer, and no text correction control appeared, so nothing could be reflected in a new reply.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Zendesk AIFailed1 of 1 test case failedCould not carry out the correction-and-re-ask task because the tested agent had no way to accept a correction.View Result →
The question was never asked again in a new conversation.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
BotpressPassed1 of 1 test case passedReflected the correction in a later answer.View Result →
The later reply dropped the assistant intro and extra return-policy block, and the deployed widget answered with the restocking-fee explanation and no preview banner.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Intercom FinPassed1 of 1 test case passedChanged the answer in a new customer conversation after the correction was saved.View Result →
The later chat gave the corrected 30-day return window and 45 days for Cedarline Plus, while the earlier answer had said 21 days from the same private sources.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (9): Ada, Chatbase, CustomGPT.ai, Decagon, FS Agent (DIY control), Gorgias, Help Scout, Respond.io, Tidio Lyro.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Feedback and learning
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →