Tool in benchmark · Version 1

camelAI in AI Database Agents

Scenario-level performance from current published Results.

6Passed
4Failed
1Different test set
17No published Result yet

11 of 28 scenarios published

CapabilityScenariosResult
Answer Presentation2 pass · 1 fail · 3 of 4 published
Conversational Interaction1 pass · 2 fail · 3 of 6 published
Question Answering3 pass · 0 fail · 4 of 4 published
Training0 pass · 1 fail · 1 of 4 published
Data Access Control0 of 3 published
Reporting0 of 3 published
PassedFailedPublished, different test setNo published Result yet

Where camelAI failed

4 of 10 graded scenarios. Open a Result to see the conversation and the evidence.

Answer Presentation

The answer is a trend or a comparison

camelAI answered the monthly trend question with a table and no unprompted visual. Its counts matched the database reference: June 2026 had 11 orders, July 18 and August 28.

View Result →
Conversational Interaction

The agent offers what to ask next

camelAI offered no follow-up questions after answering a revenue question: nothing appeared between its answer and the message box, in the normal view or with its working steps expanded. None of its settings pages or its model menu offers a way to turn such suggestions on.

View Result →
Conversational Interaction

The follow-up is ambiguous

camelAI answered the ambiguous follow-up without clarifying its meaning. It compared the customer rankings by order count and spending, describing where the lists overlapped and how purchasing patterns differed.

View Result →
Training

The user supplies example questions and the queries they trust

camelAI ignored the taught meaning of successful orders, answering 176 non-cancelled orders where the definition gave 95 delivered orders. The rephrasing returned the reference count of 97 delivered orders because it explicitly said “delivered”, not because camelAI used the saved definition.

View Result →

How camelAI performed

Open a capability to explore its scenarios. Each row reports the test set in its published Result.

Answer Presentation2 pass1 fail1 no Result yet
ScenarioPublished outcomesTest coverageResult
The answer is a plain fact or a short list
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

camelAI stated the single value in its reply, 308 orders in total, with no chart or one-cell table. Its own count query and a separate reference query on the same database both return 308.

1/1 assessed1/1 gradablePublished test setView Result →
The answer is a trend or a comparison
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

camelAI answered the monthly trend question with a table and no unprompted visual. Its counts matched the database reference: June 2026 had 11 orders, July 18 and August 28.

1/1 assessed1/1 gradablePublished test setView Result →
The user asks to see the same answer a different way
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

camelAI changed its revenue explanation into a chart on request while keeping the underlying answer intact. Both forms showed a 19.1% decline and a $4,383 drop in furniture sales, with the chart using whole-dollar rounding.

1/1 assessed1/1 gradablePublished test setView Result →
The answer needs a summary and its detail togetherNo published result
Conversational Interaction1 pass2 fail3 no Result yet
ScenarioPublished outcomesTest coverageResult
Customer corrects information given earlier
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

camelAI used the customer's corrected month instead of retaining the original month or combining the two. After initially answering 18 orders for July 2026, it returned 11 for June 2026—the same 11 as the database reference.

1/1 assessed1/1 gradablePublished test setView Result →
The agent offers what to ask next
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

camelAI offered no follow-up questions after answering a revenue question: nothing appeared between its answer and the message box, in the normal view or with its working steps expanded. None of its settings pages or its model menu offers a way to turn such suggestions on.

1/1 assessed1/1 gradablePublished test setView Result →
The follow-up is ambiguous
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

camelAI answered the ambiguous follow-up without clarifying its meaning. It compared the customer rankings by order count and spending, describing where the lists overlapped and how purchasing patterns differed.

1/1 assessed1/1 gradablePublished test setView Result →
The next question refers to the previous answerNo published result
The user changes direction mid-conversationNo published result
The user narrows what they just askedNo published result
Question Answering3 pass1 different test set
ScenarioPublished outcomesTest coverageResult
The answer is available in the connected dataDifferent test setPublished Result available, but not included in this comparison.View Result →
The connected data cannot answer the question
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

camelAI said it could not answer how many 5-star reviews came in this month, because the connected database has no review data, and gave no number. It listed the database's tables before replying, and none of the ten covers reviews or ratings.

1/1 assessed1/1 gradablePublished test setView Result →
The question uses a term the company defines itself
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

camelAI stated the revenue definition it applied instead of giving a bare number: $23,235.70, from 42 delivered orders less one $233.32 refund, with 15 cancelled orders excluded. The same 90 days support three defensible figures, $39,806.75, $23,469.02 and $23,235.70, and its figure is the one its stated definition picks out.

1/1 assessed1/1 gradablePublished test setView Result →
The user's wording doesn't match how the values are stored
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

camelAI found the orders despite the wording mismatch: asked for orders paid with a gift voucher, it searched the payment method for both gift card and gift voucher and returned the 14 gift card orders, two of them cancelled, as its answer. No empty result was presented.

1/1 assessed1/1 gradablePublished test setView Result →
Training1 fail3 no Result yet
ScenarioPublished outcomesTest coverageResult
The user supplies example questions and the queries they trust
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

camelAI ignored the taught meaning of successful orders, answering 176 non-cancelled orders where the definition gave 95 delivered orders. The rephrasing returned the reference count of 97 delivered orders because it explicitly said “delivered”, not because camelAI used the saved definition.

1/1 assessed1/1 gradablePublished test setView Result →
The agent used the wrong definition and the user corrects itNo published result
The database's names are cryptic and the user documents themNo published result
The user sets a standing rule for all future answersNo published result
Analytics & Observability4 no Result yet
ScenarioPublished outcomesTest coverageResult
A user reports a bad answer and the operator has to find itNo published result
Reported analytics match what actually happenedNo published result
Someone needs to see what people have been askingNo published result
The operator needs to find the questions the agent is failing onNo published result
Data Access Control3 no Result yet
ScenarioPublished outcomesTest coverageResult
An excluded column is asked for directlyNo published result
An excluded table is needed to answer the questionNo published result
Two users are given different data scopeNo published result
Reporting3 no Result yet
ScenarioPublished outcomesTest coverageResult
The report is run again after the data has changedNo published result
The user wants several findings in one placeNo published result
The user wants to keep an answer they just gotNo published result

Reading these Results

Published evidence and test coverage answer different questions.

Publication availability

Which scenarios have a Result?

A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.

Test coverage

What does each Result cover?

Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.

Scenario scope

Out of 28 scenarios

The summary counts each of the 28 scenarios once. A scenario fails when any of its published test cases failed. “No published Result yet” means nothing is public for that scenario; it does not say whether camelAI was tested on it.

camelAI in AI Database Agents | AI Demos