Benchmark · Version 1

AI Customer Support Chatbots

Evaluates AI support agents on grounded answers, intent understanding, conversation continuity, human handoff, connected-system actions, multilingual support, configuration and control, analytics and observability, and feedback-driven learning.

What this benchmark is

This benchmark compares AI support agents for customer-service work. It asks which products can handle real customer support conversations and connected-system actions for a business.

A reader learns how well a product answers from approved information, follows requests, keeps context, escalates properly, and supports operator control, analytics, and learning.

Every product is tested as the support agent for one fictional online store. The approved setup uses a customer-support knowledge base fixture and a connected order-system fixture, and the same implemented setup is used across products so differences come from the products, not the environment.

In scope

  • AI support chatbot or agent products purpose-built for customer support.
  • Products that answer customer questions and act on connected systems.
  • Support-agent testing for one fictional online store.
  • Handoff to human agents when the situation requires it.

Out of scope

  • General RAG chatbots with no support workflow.
  • Live chat tools with no AI.
  • Voice support agents.
  • Prompt-injection resistance.
9Capabilities
26Scenarios

Capabilities included

The broad abilities this benchmark evaluates. Each capability is defined globally; this page states that it belongs to this benchmark.

CapabilityWhat it means hereScenarios
Knowledge-grounded answeringAnswers customer questions from the company's own knowledge base, not from what the model assumes.3
Intent understandingWorks out what the customer actually wants before answering or acting.3
Conversation continuityKeeps track of what has already been said, across turns and across sessions.3
Escalation and human handoffKnows when a conversation should go to a human, and hands it over properly.3
Action executionLooks things up and makes changes in connected systems (orders, subscriptions, tickets).5
Multilingual supportHandles customers who do not write in English.2
Agent configuration and controlBehaves the way the business configures it to, with instructions applied and restrictions respected.2
Analytics and observabilityLets the operator inspect any conversation, see what the agent did, and trust the reported numbers.3
Feedback and learningAccepts corrections and improves future answers.2

Scenarios included

A scenario is a real-world situation used to test a capability. Together these scenarios define the evaluation scope of version 1.

Knowledge-grounded answering

3 scenarios
  1. Answer is available in the knowledge base
  2. Answer is not available in the knowledge base
  3. The request uses different wording than the source

Intent understanding

3 scenarios
  1. Request requires choosing the correct source or action
  2. One message contains multiple requests
  3. Request is unclear and needs clarification

Conversation continuity

3 scenarios
  1. Information from earlier in the conversation is needed later
  2. Customer corrects information given earlier
  3. Earlier conversation/session needs to be continued

Escalation and human handoff

3 scenarios
  1. User explicitly asks for a human
  2. A configured rule requires escalation
  3. Agent cannot resolve the request

Action execution

5 scenarios
  1. Retrieve information from an external system
  2. Update something in an external system
  3. Requested action cannot be completed
  4. Action requires confirmation
  5. User is not authorized to perform the action

Multilingual support

2 scenarios
  1. Customer communicates in another supported language
  2. Customer switches languages during the conversation

Agent configuration and control

2 scenarios
  1. Configured instruction changes agent behavior
  2. Configured restriction is respected

Analytics and observability

3 scenarios
  1. Conversation activity can be inspected
  2. Agent actions/tool calls can be inspected
  3. Reported analytics match what actually happened

Feedback and learning

2 scenarios
  1. Feedback can be submitted
  2. Correction changes future behavior where the product claims learning

How the benchmark works

A public summary of the evaluation method. The same defined scope and evidence standard apply to every tool assessed under this version.

Exercise-based evaluation

Evaluate exercised behaviour

The benchmark evaluates behaviour by exercising products, not by reading feature pages.

Scenario design

Make each scenario a hard condition

Each scenario is a condition that makes the capability hard, and it should be immediately understandable.

First test case

Start with the simplest direct test

The first test case under a scenario is the simplest, most natural test that directly verifies it.

Depth later

Add harder variants later

Harder variants and deeper cases are added later from observed product differences, not designed upfront.

Specification discipline

Keep literals out of design

Test cases are specifications; concrete literals are bound later at fixture implementation.

Methodology boundary

Methodology remains open

Methodology, weighting, and ranking decisions are in scope for this benchmark, but remain undesigned in the approved sources.

D4 methodology, weighting, and ranking decisions are still to do.

Resources and fixtures

The registered material and systems that create a consistent test environment for this benchmark.

Knowledge Base · v1, v2

Cedarline knowledge base (Customer Support fixture)

Knowledge base fixture used for customer-support answers and policy lookup.

API · v1

Cedarline order system API (Customer Support actions)

Order-system API fixture used for support lookups and actions.

AI Customer Support Chatbots — Benchmark definition | AI Demos