Scenario in benchmark · AI Customer Support Chatbots · Version 1

Reported analytics match what actually happened

Whether reported analytics align with the underlying recorded activity.

Capability: Analytics and observability1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the reported figures match the activity that actually occurred.
  2. Whether the result reflects the same underlying events that were recorded.
  3. Whether discrepancies between reported numbers and recorded events are exposed rather than treated as correct.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 1
BotpressFailed1 of 1 test case failedBotpress's analytics did not match what actually happened in this case.View Result →

At the same range and filters, it showed 83.8% bot assist in one place and 0% bot assisted in another, with a conflicting human-only count.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 3
Freshdesk FreddyPassed1 of 1 test case passedReported five human handoffs, matching the reference tally of five.View Result →

Its workflow totals also covered every benchmark run.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Intercom FinPassed1 of 1 test case passedIntercom Fin’s analytics matched the recorded conversation and escalation counts for this period.View Result →

Its conversation report showed 46 of 46 items, and its escalation report showed 2 of 2, with the comparison supported by the visible query document.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Zendesk AIPassed1 of 1 test case passedZendesk AI's reporting overview for 23 September to 15 October 2026 showed 50 conversations and 11 assisted escalations, and all 25 of our earlier test conversations, including the three escalation tests, appear in its conversation logs.View Result →

Its totals contain our runs; the extra conversations are other traffic on the same agent.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (10): Ada, Chatbase, CustomGPT.ai, Decagon, FS Agent (DIY control), Gorgias, Help Scout, Kommunicate, Respond.io, Tidio Lyro.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Analytics and observability

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

Reported analytics match what actually happened in AI Customer Support Chatbots | AI Demos