Scenario in benchmark · Web Search for AI · Version 1
Search only these websites
A search request is limited to a permitted set of websites.
What we check
- Whether the tool honours the restriction rather than silently ignoring it.
- Whether the tool returns relevant material from the permitted sites when such material exists.
- Whether the tool avoids bringing in results from outside the permitted set.
- Whether the tool avoids applying the restriction so aggressively that it misses usable allowed-source material.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
GPT-5.6 terraFailed1 of 1 test case failedFails this allowed-sites search task.View Result →
The outside-site page does not come back, but the answer also returns no cited page from the permitted site, so the required pair is incomplete.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
ExaPassed1 of 1 test case passedMet the site-only request.View Result →
It returned docs.github.com results, including the Copilot page at rank 2, and no GitHub blog result; a rank-5 preview-policy page mentions other dates but not the Copilot launch year.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
SerpAPIPassed1 of 1 test case passedKept the returned results within the allowed website while retaining a relevant Copilot page.View Result →
The GitHub blog announcement was absent, and the long-term support model date was not a preview launch year.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
SerperPassed1 of 1 test case passedKept the search within the permitted website and still returned a relevant page.View Result →
Its ten results came from docs.github.com, with no github.blog announcement among them.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
ValyuPassed1 of 1 test case passedHonored the website restriction while still returning relevant Copilot pages.View Result →
Its ten results came from GitHub Docs; the GitHub blog announcement containing the 2021 launch year was absent.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (7): Brave Search API, Firecrawl, Linkup, OpenAI web search, Perplexity Sonar, Tavily, You.com.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Web Retrieval
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →