Scenario in benchmark · Web Search for AI · Version 1
The request uses different wording than the source
A request uses wording different from the source, but the system must still resolve the same underlying fact or passage.
What we check
- Whether the system recognises the same underlying fact or passage despite different wording.
- Whether it relies on meaning rather than an exact phrase match.
- Whether it still resolves the correct source when the request is paraphrased.
- Whether it treats different wording as equivalent when the underlying meaning is the same.
Specific test-case inputs and grading live on the related test-case pages.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
SerpAPIFailed1 of 1 test case failedDid not retrieve an answering page for the differently worded question about keeping Netflix shares after leaving.View Result →
Its nine results covered unrelated subjects, including spelling, music videos and Istanbul cats.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Brave Search APIPassed1 of 1 test case passedFound pages answering the Netflix question despite different wording.View Result →
The first two results said the stock options are fully vested and can be kept after leaving; Netflix’s own page also returned.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
LinkupPassed1 of 1 test case passedFound pages answering the question despite the different wording.View Result →
Both search depths returned text explaining that Netflix stock options remain with employees after they leave.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
SerperPassed1 of 1 test case passedFound two pages answering the Netflix question despite different wording.View Result →
They said the shares are kept after leaving; some results gave general advice about losing unvested shares, unrelated to Netflix. Netflix's own jobs pages did not come back.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
ValyuPassed1 of 1 test case passedRetrieved an answering page despite the question using different words from the source.View Result →
Netflix's own page, ranked ninth, says employees keep their fully vested, 10-year stock options even after leaving the company.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (7): Exa, Firecrawl, GPT-5.6 terra, OpenAI web search, Perplexity Sonar, Tavily, You.com.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Web Retrieval
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →