Scenario in benchmark · AI Customer Support Chatbots · Version 1

Agent actions/tool calls can be inspected

A prior conversation contains an agent tool or action call that can be inspected.

Capability: Analytics and observability1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the earlier conversation can be found and opened.
  2. Whether an agent action or tool call in that conversation is visible for inspection.
  3. Whether the inspection shows that the call occurred, not only the final reply.
  4. Whether the call can be linked back to the conversation where it happened.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 1
KommunicateFailed1 of 1 test case failedDid not show the inspected order-lookup tool call.View Result →

In both opened conversations, the thread showed only the customer message and the reply; one reply included live order details, but no tool, function, or API entry appeared, and the events panel stayed collapsed.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 4
BotpressPassed1 of 1 test case passedLets a buyer inspect an order-lookup tool call in Monitor.View Result →

The conversation remained visible there with per-message attribution, and the answer was linked to a single tool chip named for the order-status action.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Freshdesk FreddyPassed1 of 1 test case passedPassed this inspection scenario.View Result →

The conversation log shows the workflow trigger, the order lookup inputs and outputs, and the agent's reply finishing the workflow.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Intercom FinPassed1 of 1 test case passedShows the agent’s tool call in this scenario.View Result →

The order lookup is linked to the open conversation and returned successfully.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Zendesk AIPassed1 of 1 test case passedPasses this observability check: it shows the order-lookup call and lets the operator inspect it.View Result →

The conversation lists triggered actions as successful, and opening the action details shows the action name, time, and success snapshot; the conversation is also addressable by URL.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (9): Ada, Chatbase, CustomGPT.ai, Decagon, FS Agent (DIY control), Gorgias, Help Scout, Respond.io, Tidio Lyro.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Analytics and observability

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

Agent actions/tool calls can be inspected in AI Customer Support Chatbots | AI Demos