Scenario in benchmark · Web Search for AI · Version 1
The retrieved material doesn't support an answer
Retrieved material is related to the topic but does not support an answer.
What we check
- Whether the response withholds a direct answer when the retrieved material is insufficient.
- Whether the response avoids presenting related material as if it supports the conclusion.
- Whether the response indicates that the retrieved material does not answer the question, rather than overclaiming confidence.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
TavilyFailed1 of 1 test case failedPresented approximately 366 employees as Valve Corporation's current headcount, although its retrieved material did not establish the exact current figure.View Result →
It answered rather than stating what it could not establish.
0 Pass · 1 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
FirecrawlPassed1 of 1 test case passedAcknowledged that the retrieved material did not establish Valve’s exact current employee count.View Result →
It offered a dated estimate of 384 employees rather than claiming a live total.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
SerpAPIPassed1 of 1 test case passedReturned an answer saying Valve’s exact current headcount was unavailable, while still providing labelled figures.View Result →
The relayed Google AI Overview gave a third-party estimate of 350 to 384 employees and a 2021 court-document figure of 336.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (9): Brave Search API, Exa, GPT-5.6 terra, Linkup, OpenAI web search, Perplexity Sonar, Serper, Valyu, You.com.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Answer Generation
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →