AI Demos Research — the structured-intelligence platform. Every verdict on these pages opens to the execution behind it.
Graded 16 October 2026

Does Vectara answer a naturally phrased knowledge-base question?

Vectara returned the answer from the knowledge base for this naturally phrased question. The out-of-hours passage ranked first in three independent captures, including a live notebook run, while the business-hours near-miss stayed lower each time. The corpus view shows the relevant document was present.

1 of 1 test case passed

Every test case in this scenario has a result.

Pass rate100%1 of 1 gradable
Coverage1 of 1test case with a result
1PassDid everything it was expected to do.
0FailNo test case failed.
0Not gradableEvery result here could be graded.
0UntestedEvery test case in this scenario has a result.
The pass rate is a summary. The evidence is the test case below: what we sent, what we checked, what the tool returned, and the proof.

The test case

Each test case is judged on its own: Pass, Fail, Not gradable, or Untested. The scenario result above is based on this test case.

Naturally phrased question the corpus answersPassEvidence
What we sent

“How quickly does Fenwake acknowledge a Priority 1 fault reported outside business hours?”

What the tool returned
Text
A Priority 1 fault reported outside business hours is acknowledged by a named duty engineer within 90 minutes of the ticket being raised.

Full retrieval dump: both ranking modes

Copied from Captured file (text), lines 14–29 · an excerpt

Open the whole text file ↗

Source excerpt · partial file · response 1

Text
rank=1 score=0.834820 document_id=A01_field_service_policy_and_charges_uk.pdf page=4 region=uk
Fenwake Instruments Ltd - Field Service Policy and Charges (United Kingdom) Out-of-hours cover
Outside business hours the service desk is not staffed. A duty engineer rota covers every hour of
the year,
and the out-of-hours number in section 8 pages the engineer on call. 6.1 Acknowledgement out of
hours
A Priority 1 fault reported outside business hours is acknowledged by a named duty engineer within
90
minutes of the ticket being raised. The engineer confirms ownership by telephone and, where the
customer
asks for it, in writing on the ticket. The target applies on every cover level; the cover level
determines
what happens after acknowledgement, not how quickly the call is answered. 6.2 What follows
acknowledgement
Having acknowledged, the duty engineer attempts a remote diagnosis using the telemetry record.
----------------------------------------------------------------------------------------------------

Copied from Captured file (text), lines 14–29 · an excerpt

Expected vs. Found
✓Expected: The supporting passage appears in the results.Found: It ranked first in every capture, and the business-hours near-miss stayed below it.
Supporting proof
Proof 1Screenshot
Corpus screenshot: the relevant file is present
Open original ↗
Proof 2Captured file (text)
GET capture: the supporting passage ranked first
The tool’s response is printed above, under What the tool returned. Show this file
Text62 lines
TC90 . Vectara . retrieval evidence
ClickUp: https://app.clickup.com/t/9014651757/86bbuyck7
Query: How quickly does Fenwake acknowledge a Priority 1 fault reported outside business hours?
Corpus: r6-managed-rag-ss-v1
Endpoint: GET /v2/corpora/{corpus_key}/query?query=...&limit=10
HTTP status: 200

Total results: 10
====================================================================================================
rank=1 score=0.834820 document_id=A01_field_service_policy_and_charges_uk.pdf page=4 region=uk
6.1 Acknowledgement out of hours
A Priority 1 fault reported outside business hours is acknowledged by a named duty engineer within
90
minutes of the ticket being raised.
----------------------------------------------------------------------------------------------------
rank=2 score=0.725500 document_id=A01_field_service_policy_and_charges_uk.pdf page=6 region=uk
Route     When     Detail
Service desk     Business hours     service@fenwake-instruments.example
Service desk telephone     Business hours     +44 117 496 0188
Out-of-hours pager     All other times     +44 117 496 0199
----------------------------------------------------------------------------------------------------
rank=3 score=0.721030 document_id=A01_field_service_policy_and_charges_uk.pdf page=6 region=uk
Exclusions
Part A does not apply where the instrument has been modified outside a Fenwake workshop, where a
fault
is caused by a third-party mooring or cable, or where a customer has declined a recommended firmware
update in writing.
----------------------------------------------------------------------------------------------------
rank=4 score=0.694941 document_id=A01_field_service_policy_and_charges_uk.pdf page=2 region=uk
Priority     Acknowledgement target     Measured from
Priority 1     30 minutes     Ticket raised
Priority 2     4 working hours     Ticket raised
Priority 3     The next working day     Ticket raised
----------------------------------------------------------------------------------------------------
rank=5 score=0.687505 document_id=A09_warranty_statement_service_pack.pdf page=1 region=group
Damage caused by opening the housing outside a Fenwake workshop, or by breaking a calibration seal.
----------------------------------------------------------------------------------------------------
rank=6 score=0.687505 document_id=A08_warranty_terms_sales_pack.pdf page=1 region=group
Damage caused by opening the housing outside a Fenwake workshop, or by breaking a calibration seal.
----------------------------------------------------------------------------------------------------
rank=7 score=0.686549 document_id=A01_field_service_policy_and_charges_uk.pdf page=4 region=uk
6.3 Priority 2 and Priority 3 out of hours
Priority 2 and Priority 3 tickets raised outside business hours are queued and acknowledged when the
service desk opens.
----------------------------------------------------------------------------------------------------
rank=8 score=0.680445 document_id=A09_warranty_statement_service_pack.pdf page=2 region=group
Fenwake is not liable for loss of data,
loss of survey time, vessel standby, or any other consequential loss arising from a failure, whether
or not
that failure is covered.
----------------------------------------------------------------------------------------------------
rank=9 score=0.680445 document_id=A08_warranty_terms_sales_pack.pdf page=2 region=group
Fenwake is not liable for loss of data,
loss of survey time, vessel standby, or any other consequential loss arising from a failure, whether
or not
that failure is covered.
----------------------------------------------------------------------------------------------------
rank=10 score=0.675971 document_id=A09_warranty_statement_service_pack.pdf page=1 region=group
Where a covered failure occurs,
Fenwake repairs or exchanges the instrument at its own option and bears the cost of carriage in both
directions.
----------------------------------------------------------------------------------------------------
Open the text file ↗
Proof 3Captured file (text)
POST capture: the supporting passage ranked first
The tool’s response is printed above, under What the tool returned. Show this file
Text189 lines
TC90 . Vectara . retrieval evidence
ClickUp: https://app.clickup.com/t/9014651757/86bbuyck7
Query: How quickly does Fenwake acknowledge a Priority 1 fault reported outside business hours?
Corpus: r6-managed-rag-ss-v1
Endpoint: POST /v2/corpora/{corpus_key}/query
Body: {"query": "...", "search": {"limit": 10, "reranker": {"type": "none"},
       "context_configuration": {"sentences_before": 3, "sentences_after": 3}}}
(limit=10 matches the simple-search endpoint's own default; reranker:none matches
 that same endpoint's implicit behavior, so scores below are identical to the plain
 query already reported. context_configuration adds 3 sentences of surrounding text
 on each side of the match, so each result reads as a full passage, not one sentence.)
HTTP status: 200

Total results: 10
====================================================================================================
rank=1 score=0.834820 document_id=A01_field_service_policy_and_charges_uk.pdf page=4 region=uk
Fenwake Instruments Ltd - Field Service Policy and Charges (United Kingdom) Out-of-hours cover
Outside business hours the service desk is not staffed. A duty engineer rota covers every hour of
the year,
and the out-of-hours number in section 8 pages the engineer on call. 6.1 Acknowledgement out of
hours
A Priority 1 fault reported outside business hours is acknowledged by a named duty engineer within
90
minutes of the ticket being raised. The engineer confirms ownership by telephone and, where the
customer
asks for it, in writing on the ticket. The target applies on every cover level; the cover level
determines
what happens after acknowledgement, not how quickly the call is answered. 6.2 What follows
acknowledgement
Having acknowledged, the duty engineer attempts a remote diagnosis using the telemetry record.
----------------------------------------------------------------------------------------------------
rank=2 score=0.725500 document_id=A01_field_service_policy_and_charges_uk.pdf page=6 region=uk
Fenwake internal - service operations
Page 5 Fenwake Instruments Ltd - Field Service Policy and Charges (United Kingdom) Contacts Route
When     Detail
Service desk     Business hours     service@fenwake-instruments.example
Service desk telephone     Business hours     +44 117 496 0188
Out-of-hours pager     All other times     +44 117 496 0199 Exclusions
Part A does not apply where the instrument has been modified outside a Fenwake workshop, where a
fault
is caused by a third-party mooring or cable, or where a customer has declined a recommended firmware
update in writing. In those cases the service desk will still respond, and work is quoted at the
rates in Part
B. Review
Part A is owned by the head of service operations and reviewed each spring with the regional service
leads.
----------------------------------------------------------------------------------------------------
rank=3 score=0.721030 document_id=A01_field_service_policy_and_charges_uk.pdf page=6 region=uk
Fenwake Instruments Ltd - Field Service Policy and Charges (United Kingdom) Contacts Route     When
Detail
Service desk     Business hours     service@fenwake-instruments.example
Service desk telephone     Business hours     +44 117 496 0188
Out-of-hours pager     All other times     +44 117 496 0199 Exclusions
Part A does not apply where the instrument has been modified outside a Fenwake workshop, where a
fault
is caused by a third-party mooring or cable, or where a customer has declined a recommended firmware
update in writing. In those cases the service desk will still respond, and work is quoted at the
rates in Part
B. Review
Part A is owned by the head of service operations and reviewed each spring with the regional service
leads. Fenwake internal - service operations
Page 6
----------------------------------------------------------------------------------------------------
rank=4 score=0.694941 document_id=A01_field_service_policy_and_charges_uk.pdf page=2 region=uk
Fenwake Instruments Ltd - Field Service Policy and Charges (United Kingdom) Acknowledgement during
business hours
Business hours are 08:00 to 18:00, Monday to Friday, excluding statutory non-working days in England
and Wales. Within those hours the service desk is staffed continuously and the targets below are
measured
from the moment the ticket is raised, by telephone or through the portal, whichever is earlier.
Priority     Acknowledgement target     Measured from
Priority 1     30 minutes     Ticket raised
Priority 2     4 working hours     Ticket raised
Priority 3     The next working day     Ticket raised An acknowledgement is a named engineer taking
ownership of the ticket and telling the customer what
happens next. An automatic reply from the ticketing system is not an acknowledgement and does not
stop
the clock. 4.1 What the target does not cover
The acknowledgement target is not a repair target.
----------------------------------------------------------------------------------------------------
rank=5 score=0.687505 document_id=A09_warranty_statement_service_pack.pdf page=1 region=group
2.1 Instruments repaired under warranty
A repaired instrument carries the unexpired balance of the original period, or ninety days from the
return
of the repaired instrument, whichever ends later. What is not covered
 Fouling, corrosion of sacrificial components, and wear of consumables such as anodes, guards and
     gland kits. Damage caused by a mooring failure, a vessel strike, a snagged cable or any other
mechanical event
     outside the instrument. Damage caused by opening the housing outside a Fenwake workshop, or by
breaking a calibration seal. Drift within the published specification. Drift is a calibration
matter, not a defect. Any instrument whose serial number has been removed or altered.
----------------------------------------------------------------------------------------------------
rank=6 score=0.687505 document_id=A08_warranty_terms_sales_pack.pdf page=1 region=group
2.1 Instruments repaired under warranty
A repaired instrument carries the unexpired balance of the original period, or ninety days from the
return
of the repaired instrument, whichever ends later. What is not covered
 Fouling, corrosion of sacrificial components, and wear of consumables such as anodes, guards and
     gland kits. Damage caused by a mooring failure, a vessel strike, a snagged cable or any other
mechanical event
     outside the instrument. Damage caused by opening the housing outside a Fenwake workshop, or by
breaking a calibration seal. Drift within the published specification. Drift is a calibration
matter, not a defect. Any instrument whose serial number has been removed or altered.
----------------------------------------------------------------------------------------------------
rank=7 score=0.686549 document_id=A01_field_service_policy_and_charges_uk.pdf page=4 region=uk
The target applies on every cover level; the cover level determines
what happens after acknowledgement, not how quickly the call is answered. 6.2 What follows
acknowledgement
Having acknowledged, the duty engineer attempts a remote diagnosis using the telemetry record. Where
the fault cannot be cleared remotely, attendance is arranged under section 5 from the start of the
following
working day, except on Critical cover, where the duty engineer may authorise attendance immediately.
6.3 Priority 2 and Priority 3 out of hours
Priority 2 and Priority 3 tickets raised outside business hours are queued and acknowledged when the
service desk opens. The business-hours targets in section 4 then apply from opening. Fenwake
internal - service operations
Page 4 Fenwake Instruments Ltd - Field Service Policy and Charges (United Kingdom)
----------------------------------------------------------------------------------------------------
rank=8 score=0.680445 document_id=A09_warranty_statement_service_pack.pdf page=2 region=group
Fenwake Instruments Ltd - Warranty Terms review is open. Limits
This warranty is the whole of the remedy for a defective instrument. Fenwake is not liable for loss
of data,
loss of survey time, vessel standby, or any other consequential loss arising from a failure, whether
or not
that failure is covered. Statutory rights
Nothing in these terms affects a consumer's statutory rights. The instruments described here are
sold for
commercial use, and the terms are written on that basis. Service pack
Page 2
----------------------------------------------------------------------------------------------------
rank=9 score=0.680445 document_id=A08_warranty_terms_sales_pack.pdf page=2 region=group
Sales pack
Page 1 Fenwake Instruments Ltd - Warranty Terms Limits
This warranty is the whole of the remedy for a defective instrument. Fenwake is not liable for loss
of data,
loss of survey time, vessel standby, or any other consequential loss arising from a failure, whether
or not
that failure is covered. Statutory rights
Nothing in these terms affects a consumer's statutory rights. The instruments described here are
sold for
commercial use, and the terms are written on that basis. Sales pack
Page 2
----------------------------------------------------------------------------------------------------
rank=10 score=0.675971 document_id=A09_warranty_statement_service_pack.pdf page=1 region=group
They apply to
every new Tolvan A3 supplied by Fenwake Instruments Ltd, however it was ordered and wherever it is
deployed. The warranty covers the instrument, its pressure housing, its connector and any cable
supplied with it. It
covers failure arising from a defect in materials or workmanship. Where a covered failure occurs,
Fenwake repairs or exchanges the instrument at its own option and bears the cost of carriage in both
directions. The warranty period
Every new Tolvan A3 carries a warranty of 36 months from the date the instrument is delivered. The
period runs from delivery rather than from commissioning, and is not extended by a repair carried
out
under it. 2.1 Instruments repaired under warranty
A repaired instrument carries the unexpired balance of the original period, or ninety days from the
return
of the repaired instrument, whichever ends later.
----------------------------------------------------------------------------------------------------
====================================================================================================
ADDENDUM — same query, Vectara's own default reranker enabled
Endpoint: POST /v2/corpora/{corpus_key}/query
Body: {"query": "...", "search": {"limit": 10}}  (no "reranker" key = product default,
       not the reranker:none used above)
Top 4 shown (full 10-result dump omitted here; same corpus, same passages as above)
====================================================================================================
rank=1 score=0.998649 document_id=A01_field_service_policy_and_charges_uk.pdf page=4 region=uk
6.1 Acknowledgement out of hours
A Priority 1 fault reported outside business hours is acknowledged by a named duty engineer within
90
minutes of the ticket being raised.
----------------------------------------------------------------------------------------------------
rank=2 score=0.932576 document_id=A01_field_service_policy_and_charges_uk.pdf page=4 region=uk
6.3 Priority 2 and Priority 3 out of hours
Priority 2 and Priority 3 tickets raised outside business hours are queued and acknowledged when the
service desk opens.
----------------------------------------------------------------------------------------------------
rank=3 score=0.917303 document_id=A01_field_service_policy_and_charges_uk.pdf page=2 region=uk
Priority     Acknowledgement target     Measured from
Priority 1     30 minutes     Ticket raised
Priority 2     4 working hours     Ticket raised
Priority 3     The next working day     Ticket raised
----------------------------------------------------------------------------------------------------
rank=4 score=0.612426 document_id=A01_field_service_policy_and_charges_uk.pdf page=6 region=uk
Route     When     Detail
Service desk     Business hours     service@fenwake-instruments.example
Service desk telephone     Business hours     +44 117 496 0188
Out-of-hours pager     All other times     +44 117 496 0199
----------------------------------------------------------------------------------------------------
Open the text file ↗
Proof 4Screenshot
Notebook run: HTTP 200 and rank 1
Open original ↗
Why this result

The question needed the supporting passage to surface from retrieval. That passage ranked first in three independent captures, including the live notebook run, and the business-hours near-miss stayed below it, so the right document was clearly preferred.

Tested by Vivek Tomar · evidence dated 9 September 2026

Configuration and setup

How this tool was set up for the run and what the test needed in place. Each row is a fact from the run's records; a fact the records do not hold is left out, not guessed.

Software that produced the output
Vectara
Build or version
v2
Surface
API
Set up before the run
One document answered it and no other document overlapped it, so the query was asked naturally.
Endpoint
POST /v2/corpora/{corpus_key}/query
Corpus
r6-managed-rag-ss-v1
Context window
3 sentences before/after
No reranker
reranker: {"type": "none"}
Default reranker
no "reranker" key set
Search limit
limit 10
Tested
9 September 2026 · Vivek Tomar

How this scenario is graded

How we decide Pass, Fail and Not gradable. The same rules apply to every tool tested on this scenario.

How results are decided

Each test case gets one result per tool: Pass, Fail or Not gradable. A test case we haven't run yet shows Untested. There are no partial results.

The rules
  • Pass: the tool did everything the test expected, and nothing it said contradicts the correct answer.
  • Fail: at least one expected behaviour clearly didn't happen; the test case's result says which one and quotes the tool.
  • Not gradable: our evidence couldn't settle the outcome (for example a record we needed is missing). It is never counted as a fail, and the test case's result says what's missing.

Where this sits in the benchmark

This page is one cell of a larger study: one tool, one scenario. Only this benchmark's frame appears here.

LevelNameScope
BenchmarkManaged RAG →15 scenarios · 9 tools
CapabilityRetrieval →
ScenarioAnswer is available in the knowledge base →
ToolVectara →

History of this result

What has happened to this result since it was first published. Runs and grades are never overwritten: a retest or a re-grade publishes a new result and keeps the earlier one readable.

from the publication record
5 November 2026First publishedManaged RAG v1

Act on this result

Nothing filed here edits the run or the grade. A challenge opens a review, and a review can produce a new run or a re-grade — which becomes the current result and leaves this one in the history.

This matches what I see

You run the same kind of test against your own setup and get the same behaviour.

Agree →
This does not match

Yours behaves differently. Tell us what you got, with a screenshot if you have one.

Disagree →
Point out an issue

Something here is wrong — a reference value, a transcription, a grade.

Report an issue →
Request a retest

On a newer build, a larger dataset, or your own setup.

Request a retest →
We have fixed this

Tell us what changed and we schedule a rerun of the failing test case. The old result stays as history.

Vendor notice →
Filed against this evidenceNothing yet. Challenges, counter-evidence and fix notices appear here with their outcome, and stay on the page after they are resolved.
Cite this result
aidemos.com/benchmarks/managed-rag/results/vectara/answer-is-available-in-the-knowledge-base · 1 pass · 0 fail · coverage 1/1 · graded 2026-10-16

The same record is available as structured data through the AI Demos MCP server, with the counts, the coverage and every per-test-case reason carried as fields.

Verify the proof files

These files support this result. Open a file to inspect the original evidence.

Proof 1 · Screenshot ↗image/png · 95 KB
Proof 2 · Captured file (text) ↗text/plain; charset=utf-8 · 4 KB
Proof 3 · Captured file (text) ↗text/plain; charset=utf-8 · 12 KB
Captured file (text) ↗text/plain; charset=utf-8 · 19 KB
Proof 4 · Screenshot ↗image/png · 424 KB
File fingerprints (SHA-256)

A fingerprint identifies the exact file used for this result.

Proof 1 · Screenshotf13bba325b0d2cb4b71e1eb5b98c0db99f887435d3fdc25c45c8e3fbee2b39e0
Proof 2 · Captured file (text)b36685fc12696b1f3db4295638ca5bb4b9d2b1659759eefc739b99852dbf49a8
Proof 3 · Captured file (text)3de3950f0fdda7a0dab3882040336e6f8aebc00a71c9a03f7101fc58942a37f9
Captured file (text)d23683233dca7925307f7cc57e4d09835b76c9601c297cb03ba203dfb6c2e674
Proof 4 · Screenshotd5d9f33015af9cfe4fa2b11112a1c5d2ee78739f3901382dcf524f4a6d230a43