Scenario in benchmark · Web Search for AI · Version 1

A page on the web answers the question

A public web page directly supports the answer.

Capability: Web Retrieval1 pinned test case per toolNeutral global definition →

What we check

  1. Whether it locates a public page that directly supports the answer.
  2. Whether it uses that support when forming the response.
  3. Whether it treats the case as answered by public evidence rather than as unsupported.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Passed · 4
ExaPassed1 of 1 test case passedAnswered the question by returning pages that state the Files API upload limit is 512 MB.View Result →

The OpenAI Files API reference is at rank 1, and all ten returned results carry that figure; the other sizes mentioned on those pages are different limits.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

LinkupPassed1 of 1 test case passedFound a page answering the question: OpenAI’s Files API reference at rank 8 stated a maximum individual-file upload size of 512 MB.View Result →

In all, seven of the ten standard-mode results contained that figure.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

SerperPassed1 of 1 test case passedPassed this scenario: it returned a page that answered the question.View Result →

The search results included the Files API page stating the per-file limit is 512 MB, and five other results also showed that figure.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

ValyuPassed1 of 1 test case passedRetrieved a page answering the OpenAI Files API upload-limit question with a limit of 512 MB per file.View Result →

The answer appeared in the returned page text, and eight of ten results contained that figure.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (8): Brave Search API, Firecrawl, GPT-5.6 terra, OpenAI web search, Perplexity Sonar, SerpAPI, Tavily, You.com.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Web Retrieval

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

A page on the web answers the question in Web Search for AI | AI Demos