Scenario in benchmark · Web Search for AI · Version 1

Something more famous has the same name

A query targets a less famous entity that shares its name with a more famous one.

Capability: Web Retrieval1 pinned test case per toolNeutral global definition →

What we check

  1. Whether the system recognises the intended entity rather than defaulting to the more famous namesake.
  2. Whether it avoids confidently returning results about the wrong same-named entity.
  3. Whether it keeps retrieval focused on the less famous target when the namesake is the stronger public association.

Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

How the tools performed

One row per tool, from its current published Result for this scenario and benchmark version. Open a row for the full summary and the exact counts.

Passed · 7
ExaPassed1 of 1 test case passedReturned ten pages about Grafana Labs’ Loki and none about Marvel’s Loki.View Result →

The returned documentation also identified the required compactor configuration block for enabling retention.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

FirecrawlPassed1 of 1 test case passedReturned ten results, all about Grafana Labs' Loki log system and none about the far more famous Marvel namesake.View Result →

The top result, Loki's official documentation, says retention works through the Compactor and that the retention setting must be turned on; the second result shows that setting under the compactor block.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

LinkupPassed1 of 1 test case passedKept the search focused on Grafana Loki rather than its Marvel namesake.View Result →

All ten results covered log retention and named the compactor block.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

SerpAPIPassed1 of 1 test case passedReturned results about Grafana’s Loki log system rather than its more famous Marvel namesake.View Result →

The retrieved documentation also supplied the configuration detail needed to delete old logs.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

SerperPassed1 of 1 test case passedFound the intended subject despite its more famous namesake.View Result →

All ten results concerned Grafana Labs’ Loki, and snippets at ranks 1, 4 and 10 pointed to the compactor.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

ValyuPassed1 of 1 test case passedReturned ten results about Grafana Loki, not its Marvel namesake.View Result →

Nine mentioned the compactor.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

You.comPassed1 of 1 test case passedReturned ten results for a question about Loki, the log aggregation system from Grafana Labs, and every one was about that Loki.View Result →

None was about the Marvel character.

1 Pass · 0 Fail · 0 Not gradable · 1 of 1 assessed · 1 of 1 gradable

Failures first, then A–Z within each group. Not a ranking. Assessed = Pass + Fail + Not gradable; Gradable = Pass + Fail, both out of the Result’s pinned test cases.

No published Result yet (5): Brave Search API, GPT-5.6 terra, OpenAI web search, Perplexity Sonar, Tavily.

No published Result does not tell you whether a tool has been tested.

Test design

Pinned test cases
1
Disclosure
0 public · 1 withheld
Capabilities exercised here
Web Retrieval

How the results are graded

  • Pass the test-case expectations hold.
  • Fail an expectation demonstrably does not hold.
  • Not gradable the evidence cannot establish the outcome.

Version 1 uses test-case expectations; no scenario rubric is pinned. Benchmark methodology →

Something more famous has the same name in Web Search for AI | AI Demos