Vectorize.io in Managed RAG
Scenario-level performance from current published Results.
11 of 15 scenarios published
Where Vectorize.io failed
2 of 11 graded scenarios. Open a Result to see the conversation and the evidence.
A claim in the answer traced back to its source
Vectorize.io failed to give a citation that opened to text supporting the answer's claim. Both executions showed no citation pointer, so nothing could be opened against the claim.
View Result →The retrieved sources disagree with each other
Vectorize.io presented a single warranty period as settled fact rather than acknowledge conflicting sources. A separate retrieval request for the same question returned 24-month and 36-month warranties as its top two results; the answer gave 24 months from delivery.
View Result →How Vectorize.io performed
Open a capability to explore its scenarios. Each row reports the test set in its published Result.
Citations1 fail
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| A claim in the answer traced back to its source | 0 Pass1 Fail0 Not gradable 1 of 1 test case failedWhat happenedVectorize.io failed to give a citation that opened to text supporting the answer's claim. Both executions showed no citation pointer, so nothing could be opened against the claim. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Data Ingestion3 pass1 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| A cleanly scanned document | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedVectorize.io turned a one-page scanned calibration certificate, whose text existed only as an image, into readable text and returned it as the top passage for a question about it. That passage gives the as-found bias as -0.037 psu, the figure on the certificate. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| A simple, clearly formatted table | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedVectorize.io correctly read £487 from the Extended cover row and out-of-hours callout column. Its highest-ranked passage contained the whole charges table, and its generated answer gave that amount. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| An ordinary digital text document | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedVectorize.io returned the passage stating the asked fact as its top result when the question used the document's own words. Of the five results, the Tolvan A3 product overview passage giving the 22-month recalibration interval ranked first, with a relevancy score of 1.0000. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| A bad file in the upload | No published result | ||
Grounded Generation1 pass1 fail1 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| The retrieved material doesn't support an answer | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedVectorize.io declined to answer a question its documents did not cover. Asked how much paid parental leave Fenwake gives, two separate answers each said the documents provided don't specify it, gave no figure or Fenwake policy, and pointed the user to HR or an updated employee handbook. Retrieval for the same question returned nothing on parental leave. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The retrieved sources disagree with each other | 0 Pass1 Fail0 Not gradable 1 of 1 test case failedWhat happenedVectorize.io presented a single warranty period as settled fact rather than acknowledge conflicting sources. A separate retrieval request for the same question returned 24-month and 36-month warranties as its top two results; the answer gave 24 months from delivery. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The retrieved material supports an answer | No published result | ||
Knowledge Sync1 pass2 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| New content is added to the source | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedVectorize.io made newly added content findable after the update path finished. Before the add, the question scored almost zero; after the refresh, retrieval returned the added bulletin and the answer gave October 14, 2026. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Content is removed from the source | No published result | ||
| Existing content is changed | No published result | ||
Retrieval4 pass
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Answer is available in the knowledge base | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedVectorize.io retrieved the knowledge-base passage answering Fenwake’s out-of-hours Priority 1 question. The passage ranked highest by relevance and specified acknowledgement within 90 minutes. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Answer is not available in the knowledge base | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedVectorize.io signalled that it had nothing for a question its documents do not answer, through low relevance scores rather than an explicit no-match. None of the five passages it returned answers the parental-leave question; the best relevance score was 0.3473 (average 0.0726), against 0.9999542 (average 0.7537) for an answerable question on the same documents. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Retrieval is scoped to part of the documents | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedVectorize.io returned only Nordic passages when the search was limited to Nordic-region documents, leaving out the UK document containing the answer. The same search without the region limit did return that document. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The request uses different wording than the source | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedVectorize.io returned the passage that answers a question worded differently from its documents. Asked about "paid time off", a term the documents never use, it ranked first the employee handbook clause giving the 26-day annual leave entitlement, far above unrelated results. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Reading these Results
Published evidence and test coverage answer different questions.
Which scenarios have a Result?
A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.
What does each Result cover?
Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.
Out of 15 scenarios
The summary counts each of the 15 scenarios once. A scenario fails when any of its published test cases failed. “No published Result yet” means nothing is public for that scenario; it does not say whether Vectorize.io was tested on it.