Scenario in benchmark · Web Search for AI · Version 1
Something more famous has the same name
A query targets a less famous entity that shares its name with a more famous one.
What we check
- Whether the system recognises the intended entity rather than defaulting to the more famous namesake.
- Whether it avoids confidently returning results about the wrong same-named entity.
- Whether it keeps retrieval focused on the less famous target when the namesake is the stronger public association.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the tools performed
One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.
ExaPassed1 of 1 test case passedReturned ten pages about Grafana Labs’ Loki and none about Marvel’s Loki.View Result →
The returned documentation also identified the required compactor configuration block for enabling retention.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
FirecrawlPassed1 of 1 test case passedReturned ten results, all about Grafana Labs' Loki log system and none about the far more famous Marvel namesake.View Result →
The top result, Loki's official documentation, says retention works through the Compactor and that the retention setting must be turned on; the second result shows that setting under the compactor block.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
LinkupPassed1 of 1 test case passedKept the search focused on Grafana Loki rather than its Marvel namesake.View Result →
All ten results covered log retention and named the compactor block.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
SerpAPIPassed1 of 1 test case passedReturned results about Grafana’s Loki log system rather than its more famous Marvel namesake.View Result →
The retrieved documentation also supplied the configuration detail needed to delete old logs.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
SerperPassed1 of 1 test case passedFound the intended subject despite its more famous namesake.View Result →
All ten results concerned Grafana Labs’ Loki, and snippets at ranks 1, 4 and 10 pointed to the compactor.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
ValyuPassed1 of 1 test case passedReturned ten results about Grafana Loki, not its Marvel namesake.View Result →
Nine mentioned the compactor.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
You.comPassed1 of 1 test case passedReturned ten results for a question about Loki, the log aggregation system from Grafana Labs, and every one was about that Loki.View Result →
None was about the Marvel character.
1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable
Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.
No published Result yet (5): Brave Search API, GPT-5.6 terra, OpenAI web search, Perplexity Sonar, Tavily.
No published Result does not tell you whether a tool has been tested.Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Web Retrieval
How the results are graded
- Pass the test-case expectations hold.
- Fail an expectation demonstrably does not hold.
- Not gradable the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →