Vectara in Managed RAG
Scenario-level performance from current published Results.
13 of 15 scenarios published
Vectara passed every graded scenario it has a published Result for.
How Vectara performed
Open a capability to explore its scenarios. Each row reports the test set in its published Result.
Citations1 pass
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| A claim in the answer traced back to its source | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedVectara cited a passage that supported its answer's claim. It said a named duty engineer would acknowledge the out-of-hours fault within 90 minutes, and the cited passage said the same. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Data Ingestion3 pass1 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| A bad file in the upload | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedVectara reported files it could not process during upload. It returned errors for a truncated PDF and a password-protected PDF; the console also showed the truncated file's failure. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| A simple, clearly formatted table | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedVectara correctly answered the row-and-column lookup with 487 for an Extended contract’s out-of-hours callout charge. The answer cited the passage containing the charges table. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| An ordinary digital text document | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedVectara returned the requested fact in the document's own words. In both configurations, the top result was the passage stating a 22-month recalibration interval, and the next results were different warranty passages; both captures showed HTTP 200. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| A cleanly scanned document | No published result | ||
Grounded Generation3 pass
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| The retrieved material doesn't support an answer | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedVectara declined to answer a question its retrieved material did not support: it said it did not have enough information about how much paid parental leave Fenwake gives and offered no figure or policy. None of the ten passages it retrieved mentions parental leave. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The retrieved material supports an answer | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedVectara gave the supported 90-minute acknowledgement time for an out-of-hours fault. It did not mix in the competing 30-minute target from another retrieved passage. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The retrieved sources disagree with each other | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedVectara said it could not determine the warranty period on a new Tolvan A3 because its search results conflict, and named neither 24 nor 36 months as the fact. Both conflicting warranty documents were among the passages it retrieved, at ranks 1 and 2. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Knowledge Sync2 pass1 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Content is removed from the source | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedVectara stopped using the deleted source and declined to answer the question. Across four repeat queries, the deleted document was absent from the top ten results and the supplier was not cited. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| New content is added to the source | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedVectara made a newly added document findable right away: before the add no retrieved passage mentioned the Tromso depot, and afterwards all four identical queries retrieved the passage giving its opening date. Only two of those four answers stated the date; the other two declined. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Existing content is changed | No published result | ||
Retrieval4 pass
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Answer is available in the knowledge base | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedVectara returned the answer from the knowledge base for this naturally phrased question. The out-of-hours passage ranked first in three independent captures, including a live notebook run, while the business-hours near-miss stayed lower each time. The corpus view shows the relevant document was present. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Answer is not available in the knowledge base | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedVectara signaled the unavailable paid-parental-leave answer through clearly lower relevance scores. In a second default-reranker call with no context window specified, the top score was 0.0820, versus 0.9986 for an answerable question on the same documents. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Retrieval is scoped to part of the documents | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedVectara kept retrieval within the requested document subset without returning an empty result. The document containing the answer ranked first in an unrestricted search and was absent from the restricted results. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The request uses different wording than the source | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedVectara returned the matching passage even though the question used different wording from the source. In both retrieval runs, the annual leave passage was rank 1, and the returned passages did not contain the query's alternative terms. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Reading these Results
Published evidence and test coverage answer different questions.
Which scenarios have a Result?
A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.
What does each Result cover?
Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.
Out of 15 scenarios
The summary counts each of the 15 scenarios once. A scenario fails when any of its published test cases failed. “No published Result yet” means nothing is public for that scenario; it does not say whether Vectara was tested on it.