Scenario in benchmark · Web Search for AI · Version 1
A new page has just gone live
A page has just gone live, and the tool should be able to find it.
What we check
- Whether the required before-check confirms the page was not already findable.
- Whether the tool finds the newly published page once it is available.
- Whether a run that starts with the page already findable is treated as invalid, not as a freshness success.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
ExaPassed1 of 1 test case passedExa's search service found the newly launched Ostrell oak sideboard page at rank 1, with its $1185.00 price, about 40 hours after it went live.View Result →
Before launch it found nothing from the store, and two searches about 4.5 and 4.7 hours after launch returned no page from the store either.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
SerpAPIPassed1 of 1 test case passedFound the newly published product page between 40 and 42 hours after publication.View Result →
Its organic search placed the page first and included the store page’s $1185.00 price.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
SerperPassed1 of 1 test case passedFound the live Cedarline page with its $1185.00 price.View Result →
The page went live on 21 Oct at 16:16 UTC; Serper found it at rank 1 about 40 hours later. The 17 Oct search and the 21 Oct searches at 20:46:50 UTC and 21:01:18 UTC found no Cedarline page or that price; the successful search was on 23 Oct at 08:31:43 UTC.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (9): Brave Search API, Firecrawl, GPT-5.6 terra, Linkup, OpenAI web search, Perplexity Sonar, Tavily, Valyu, You.com.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Freshness
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →