Scenario in benchmark · Web Search for AI · Version 1

The request uses different wording than the source

A request uses wording different from the source, but the system must still resolve the same underlying fact or passage.

Capability: Web Retrieval1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the system recognises the same underlying fact or passage despite different wording.
  2. Whether it relies on meaning rather than an exact phrase match.
  3. Whether it still resolves the correct source when the request is paraphrased.
  4. Whether it treats different wording as equivalent when the underlying meaning is the same.

Specific test-case inputs and grading live on the related test-case pages.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Failed · 1
SerpAPIFailed1 of 1 test case failedDid not retrieve an answering page for the differently worded question about keeping Netflix shares after leaving.View Result →

Its nine results covered unrelated subjects, including spelling, music videos and Istanbul cats.

0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Passed · 4
Brave Search APIPassed1 of 1 test case passedFound pages answering the Netflix question despite different wording.View Result →

The first two results said the stock options are fully vested and can be kept after leaving; Netflix’s own page also returned.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

LinkupPassed1 of 1 test case passedFound pages answering the question despite the different wording.View Result →

Both search depths returned text explaining that Netflix stock options remain with employees after they leave.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

SerperPassed1 of 1 test case passedFound two pages answering the Netflix question despite different wording.View Result →

They said the shares are kept after leaving; some results gave general advice about losing unvested shares, unrelated to Netflix. Netflix's own jobs pages did not come back.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

ValyuPassed1 of 1 test case passedRetrieved an answering page despite the question using different words from the source.View Result →

Netflix's own page, ranked ninth, says employees keep their fully vested, 10-year stock options even after leaving the company.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (7): Exa, Firecrawl, GPT-5.6 terra, OpenAI web search, Perplexity Sonar, Tavily, You.com.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Web Retrieval

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

The request uses different wording than the source in Web Search for AI | AI Demos