Does SerpAPI return a page that answers the question when the question uses different wording?
SerpAPI did not retrieve an answering page for the differently worded question about keeping Netflix shares after leaving. Its nine results covered unrelated subjects, including spelling, music videos and Istanbul cats.
1 of 1 test case failed
Every test case in this scenario has a result.
The test case
Each test case is judged on its own: Pass, Fail, Not gradable, or Untested. The scenario result above is based on this test case.
The question uses different words than the pageFailEvidence
“If I quit Netflix, do I lose the shares I was granted while I was working there?”
{ "results": [ { "rank": 1, "url": "https://www.reddit.com/r/ENGLISH/comments/1mtxpi2/their_and_there/", "title": "Their and there : r/ENGLISH", "content": "I still do not know how to use \"their\" and \"there\". \"Is there a reason for that?\" is that correct or what?", "content_chars": 106, "published_date": "Aug 18, 2025", "url_resolved_from": "link" }, { "rank": 2, "url": "https://x.com/mitchellh/status/2055380239711457578", "title": "Mitchell Hashimoto on X", "content": "The main issue is I don't even know how to bring this up to people I know personally, because bringing this topic up leads to immediately ...", "content_chars": 141, "published_date": "May 15, 2026", "url_resolved_from": "link" }, { "rank": 3, "url": "https://paulgraham.com/goodtaste.html", "title": "Is There Such a Thing as Good Taste?", "content": "If we start from the premise that there's no such thing as good taste, The solution to this puzzle is to realize that the purpose of art is to work on its ...", "content_chars": 158, "published_date": null, "url_resolved_from": "link" }, { "rank": 4, "url": "https://www.youtube.com/watch?v=z-xX54RkJjw", "title": "Till There Was You ⎪Joscho Stephan Trio", "content": "Learn gypsy jazz guitar with Joscho Stephan! Join his official online academy for lessons, playalongs. Till There Was You - Lesson", "content_chars": 130, "published_date": "Jun 20, 2020", "url_resolved_from": "link" }, { "rank": 5, "url": "https://www.youtube.com/watch?v=sLYfZTOW9Lc", "title": "Zach Williams - There Was Jesus (Live From Red Rocks)", "content": "Watch & share my official music video for “There Was Jesus (Live)” now! Listen to my latest release here - https://ZachWilliams.l.", "content_chars": 130, "published_date": "Jun 4, 2026", "url_resolved_from": "link" }, { "rank": 6, "url": "https://www.legalnomads.com/istanbul-cats/", "title": "Why Are There So Many Cats in Istanbul? - Legal Nomads", "content": "After visiting Istanbul, I wondered why there were so many cats in town. This piece dives into the history & culture of stray cats in Turkey.", "content_chars": 141, "published_date": null, "url_resolved_from": "link" }, { "rank": 7, "url": "https://orionmagazine.org/article/six-questions-for-the-author-once-there-were-wolves/", "title": "Six Questions for the Author: Once There Were Wolves", "content": "CHARLOTTE MCCONAGHY'S Once There Were Wolves, set in the Scottish Highlands, follows Inti Flynn, a scientist dedicated to reintroducing wolves ...", "content_chars": 146, "published_date": "Aug 4, 2021", "url_resolved_from": "link" }, { "rank": 8, "url": "https://sophiebits.com/2026/06/25/there-are-no-lossless-transformations-of-natural-language-text", "title": "There are no lossless transformations of natural-language ...", "content": "If you circumvent this process, you will probably walk away with a poorer understanding of the subject matter.", "content_chars": 110, "published_date": "Jun 25, 2026", "url_resolved_from": "link" }, { "rank": 9, "url": "https://georgerrmartin.com/notablog/2024/07/11/here-there-be-dragons-2/", "title": "Here There Be Dragons | Not a Blog", "content": "“The Red Dragon and the Gold,” the fourth episode of season 2 of HOUSE OF THE DRAGON. this episode delivered it in spades with the Battle of ...", "content_chars": 144, "published_date": "Jul 11, 2024", "url_resolved_from": "link" } ] }
API exchange: the question and nine unrelated search results
Copied from Captured file (JSON), results
Open the whole JSON file ↗| ✗ | Expected: Return a page that genuinely answers the question despite using different wording.Found: The returned pages did not discuss Netflix, shares, equity or stock options, and none answered the question. |
- request
- { "engine": "google_ai_overview", "page_token": "QFM4dHiczVXbcqpIFP2A-Y7xxZSAaI6mikp1t4hoNMF4iXmxuENQWmkIIW_zL_M2fzRfMw2iQoLJnPMwNbsqabpZ7t63xfrjz43re3__9pcThjtywzBxHDdsjO2N2dDxllFJ4uuMhTck0J3b5FXgazoR2JrpChxAoRqMBsiQgvYEPyThdRABuZb9RFhrKjFfqMe3F8KsGfrP-53v0U2DNEy_Qd4Sn7elWDZwo8k2-cY9o24p … [1,391 characters in the file]" }
- response › search_information
- { "ai_overview_state": "Fully empty" }
Show the whole file (7 KB)
{ "request": { "engine": "google_ai_overview", "page_token": "QFM4dHiczVXbcqpIFP2A-Y7xxZSAaI6mikp1t4hoNMF4iXmxuENQWmkIIW_zL_M2fzRfMw2iQoLJnPMwNbsqabpZ7t63xfrjz43re3__9pcThjtywzBxHDdsjO2N2dDxllFJ4uuMhTck0J3b5FXgazoR2JrpChxAoRqMBsiQgvYEPyThdRABuZb9RFhrKjFfqMe3F8KsGfrP-53v0U2DNEy_Qd4Sn7elWDZwo8k2-cY9o24pAPx3Jio2TNchQJdB9JX4vavUgQygWPAkAztb4Rm1OrxGSvYK2Re9wcOd0wyu0L_e-n0Q9-hpXp_exd_SK2BpG2exABBnUULgfp9NwZQ8l2NeX1TqhC5bXj18Pin2uBRryey4tO3TKNYAofMlxcuoG6cUxqlSug06Awx7EMwvx03rM5JB2wOKWJVC0RRx0Tp17m4l0RPGsOngsowW0IVjXN3Rs4eApPOMZl0W99v69HHxtIj28LnZHP94e_SeEn8fqz3LZnSN4jZYNUxj7azZ22zPNbk22-V_dK4yIunkxCRSZNI9XnWip2e-47Uadw3IHWmEDk0Hnzom0j2SKxKDKIdWtLgw2GnD8pYq_aLfvJUQIAzEKSi3dvSQ1lU69ozO6DttTRFxvjUd_s80OphdAFMaZYHIOAcWEQeUVKBuobEyqPrMfMxbGZwfe5-AJQIimq8HjoVZffS8AgqyxZQDSBzAvGp60QfNeHTawGPAIqBfKHhEHAMUc1AhprQEsgLGQBapT3cuu3bqXBEPgabDkN_VFz8w7sLAr-g1S_RvPn0VBr8jUWrKkS8fxr9IG7EDJvKze6fByND068fJpPuwMmW48CZxNU3wVvsJxdH_V0Q66JGY69FlIonfEynTI_iFHmXPZz1Khxfhi0SCUrUejSv1qJw3AtMikb7Wo58mEvqSSIcqVRIJVxNpd5FIdvyRSGU9Eqv16ESkX9Ej-It6lPHKPCxlcYpe7IsiJUVE81u9RfvagJ0ZIoPBW4t3VK5_B8C3ImVtw5utamyu1q5xA5vDMDGu1vQA35hGbS_IVl2u7yM3rE_M0Nq4b1d1A9OjDSZmPXTMOnHUwCT0JFZJ3Q5UPzSNeuy4GzM_i3Hgub6dggPztrbdBFhg1VbrbvrAStuX8XCi6O2kJ3XZqcwuwn3YVMbP2r3m7mxl5bsReNdDjnf5bjQXlf3sdei8JlGL6yeL5TuPHOyvYPK6f1n2n5c7MJqtWuBhPpsPtUEn0UJsJr2WpSXN5SgaqWMLexPt0QGxE2zjydbv7puz3Ww48qeWl0VGBEnl7lXPelqM6EmgC3RQWjaYKGBp04lF2SSDuYrpGgNFuqaNhDWiq2uTvApcx9L1rsV3ul2N583r2tYOk53QZWvEDwTuHwBMT2Y" }, "http_status": 200, "latency_ms": 1396.9, "timestamp_utc": "2026-10-03T12:09:14Z", "response": { "search_metadata": { "id": "6ac0f0693ee5df47e959aa97", "status": "Success", "json_endpoint": "https://serpapi.com/searches/_qT4LPnex-eNf6mMAntv9QSvYsobfceRQUjMp3maLBw/6ac0f0693ee5df47e959aa97.json", "markdown_endpoint": "https://serpapi.com/searches/_qT4LPnex-eNf6mMAntv9QSvYsobfceRQUjMp3maLBw/6ac0f0693ee5df47e959aa97.md", "created_at": "2026-10-03 12:09:13 UTC", "processed_at": "2026-10-03 12:09:13 UTC", "google_ai_overview_url": "https://www.google.com/async/folsrch?yv=3&cs=0&ei=1ACtarKHCdGr5NoPyt6ruAI&async=_basejs:/xjs/_/js/k%3Dxjs.s.en.sxyn3gGwIdo.2023.O/am%3DAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEQgBAAAAJACAAAAAAAAAAAAAAAAAAAAAAAAAACAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAAIABEACAAAAAAAAAIAgAAAAABAAAAAAAAAAYACAAAACQAgAAACgAAAAAAAAAAAAAAAAAAAAAAAACBAEAAAARCAAAAQAAAD_zHwDAEADAAAAAADgAAAAAAAAAAAAAAAAAAAAAAAAACABAAAAAAAAAAAACwAAAIAAAwAACAABAiAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAIACAAAAAAACAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAEAAAAAAoAAAAAAAAAADAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAgwAAAAAAAAAAAAAFAAI_ACCAAQAAAAAAAQAAAAAAAAAAABAAhAAAAAAAAAAQAAEADAAAAAAcgA8HoBDBAUAAAAAAAAAAAAAAAAAAAAAAAAAAIAAKIA5kAQEQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQEV4CgAAAAAAALYGAAQ/dg%3D0/br%3D1/ichc%3D1/rs%3DACT90oF5cRSVXVuqBZ22M7xSkXynqwaDfg/cb%3Dloaded_h_0?cb%3D121509378,_basecss:/xjs/_/ss/k%3Dxjs.s.OoY8uXZ38k4.L.B1.O/am%3DCAAIAAAAAACAAAAAAAAAAAAAAECAACIAAAAAAAAAAAAAAAAAAABCACAAAAACAAAAAAAAAAAAAAAAAABEACAAAAAAABAAAEAAAAAAQFAAAAAAAAAAAEABAAAAABACoAERAAABAAAAAAAAAAAKPQAAAGAQAAAAAACABzAAcBAAAAAAAAAAACAAAAAAAAABAEACAAAAAAAAAIAgAAAAAAAAAAAgAAAAAACAAAACQAEAAAAIoAAAAAIAAAAAAgAAAAAAAAAACAGEAAAAAAAAAAQAAAAAAAAAAIADAAAAAAAAAAAAAAAAAAACAAAAAAAAAAAAAAAQHAAAAAAAAAQDAAAAAAAAAAAACAAAAgAAAAAAAAAAAACRAAkAAAEAAAAAYAAAAAAAAAAAAAAAAYAQCgEAAoACEHBFAAAAAAcAAgAAAAAAAAAEACKAAAAAAAAABAAAAAAQAEAQgBBAAAAAAEAACAAAAAAEAAAAABADAAAAAAAAAAIoAAIQAMAIEgAAiUIigAgAAQEAAAgAACAAIgAAAAAAFEBAAAAAAAAAAAAAAEQAAAAAAAAAAAAAAAAAAAAAYAEAWCAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQ/br%3D1/cb%3Dloaded_h_0/rs%3DACT90oE8ANIZiLbBudbc6SNN9PYeIBVkNw?cb%3D121509378,_basecomb:/xjs/_/js/k%3Dxjs.s.en.sxyn3gGwIdo.2023.O/ck%3Dxjs.s.OoY8uXZ38k4.L.B1.O/am%3DCAAIAAAAAACAAAAAAAAAAAAAAECAACIAAAAAAAAAAAAAAAAAAABCACAAAAACAAAAAAAAAAAAAAAAAABEACAAAAAAABAAAEAAAAAAQFAAAAAEQgBAAEAJACAAABACoAERAAABAAAAAAAAAAAKPQAEAGAQAAAAAACABzAAcBAAAAAAAAAAACAAAACAAAIBBEACAAAAAAAAAIAgAAAAABAAAAAgAAAAYACAAAACQAkAAACooAAAAAIAAAAAAgAAAAAAAAAACBGEAAAARCAAAAQAAAD_zHwDAMADAAAAAADgAAAAAAAAAAACAAAAAAAAAAAAACARHAAAAAAAAAQDCwAAAIAAAwAACAABAiAAAAAAAAAAAACRAAkAAAEAAAAAYAAAAAAAAAAAAAAAAYAQCgEAAoACEHBFAAACAAcAAgAAAAAAAAAEACKAAAAAAAAABQAAAAAQAEAQgBBAAAAAAEAACAAAoAAEAAAAABADAAAAAAAAAAIpAAIQAMAIEgAAiUIigAgAAQEAAgwAACAAIgAAAAAAFEBI_ACCAAQAAAAAAEQAAAAAAAAAAABAAhAAAAAAYAEAWCAEADAAAAAAcgA8HoBDBAUAAAAAAAAAAAAAAAAAAAAAAAAAAIAAKIA5kBQEQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQEV4CgAAAAAAALYGAAQ/d%3D1/ed%3D1/dg%3D0/br%3D1/ujg%3D1/ichc%3D1/rs%3DACT90oGusbn4DV56dB8TCsHHx43ha1FLAA/cb%3Dloaded_h_0?cb%3D121509378,_fmt:madl,_id:B2Jtyd,_mado:ed&q=If+I+quit+Netflix,+do+I+lose+the+shares+I+was+granted+while+I+was+working+there?&mlro=0a44LRP0GmjMJNQc5yDG90RI0Vtqt2QMZbObipgQYniuAzct13i39uUEQqTvJhvyu41FyVWz3ChonYByvqjWFZWpAKTY4APUTUJbH8ybtoeyD4fby2WKuKaMfokNbShAwhrmwNmn9q2TpTJKnRfk&mlros=Ga1OakfXVKk&mrc=CAA4gANQAWgAcACYAQCgAUaoAQCwAQG6AQEB&sca_esv=18fcc9f3899b33e6&mgtyp=90&snr=1", "raw_html_file": "https://serpapi.com/searches/_qT4LPnex-eNf6mMAntv9QSvYsobfceRQUjMp3maLBw/6ac0f0693ee5df47e959aa97.html", "total_time_taken": 0.85 }, "search_parameters": { "engine": "google_ai_overview", "page_token": "QFM4dHiczVXbcqpIFP2A-Y7xxZSAaI6mikp1t4hoNMF4iXmxuENQWmkIIW_zL_M2fzRfMw2iQoLJnPMwNbsqabpZ7t63xfrjz43re3__9pcThjtywzBxHDdsjO2N2dDxllFJ4uuMhTck0J3b5FXgazoR2JrpChxAoRqMBsiQgvYEPyThdRABuZb9RFhrKjFfqMe3F8KsGfrP-53v0U2DNEy_Qd4Sn7elWDZwo8k2-cY9o24pAPx3Jio2TNchQJdB9JX4vavUgQygWPAkAztb4Rm1OrxGSvYK2Re9wcOd0wyu0L_e-n0Q9-hpXp_exd_SK2BpG2exABBnUULgfp9NwZQ8l2NeX1TqhC5bXj18Pin2uBRryey4tO3TKNYAofMlxcuoG6cUxqlSug06Awx7EMwvx03rM5JB2wOKWJVC0RRx0Tp17m4l0RPGsOngsowW0IVjXN3Rs4eApPOMZl0W99v69HHxtIj28LnZHP94e_SeEn8fqz3LZnSN4jZYNUxj7azZ22zPNbk22-V_dK4yIunkxCRSZNI9XnWip2e-47Uadw3IHWmEDk0Hnzom0j2SKxKDKIdWtLgw2GnD8pYq_aLfvJUQIAzEKSi3dvSQ1lU69ozO6DttTRFxvjUd_s80OphdAFMaZYHIOAcWEQeUVKBuobEyqPrMfMxbGZwfe5-AJQIimq8HjoVZffS8AgqyxZQDSBzAvGp60QfNeHTawGPAIqBfKHhEHAMUc1AhprQEsgLGQBapT3cuu3bqXBEPgabDkN_VFz8w7sLAr-g1S_RvPn0VBr8jUWrKkS8fxr9IG7EDJvKze6fByND068fJpPuwMmW48CZxNU3wVvsJxdH_V0Q66JGY69FlIonfEynTI_iFHmXPZz1Khxfhi0SCUrUejSv1qJw3AtMikb7Wo58mEvqSSIcqVRIJVxNpd5FIdvyRSGU9Eqv16ESkX9Ej-It6lPHKPCxlcYpe7IsiJUVE81u9RfvagJ0ZIoPBW4t3VK5_B8C3ImVtw5utamyu1q5xA5vDMDGu1vQA35hGbS_IVl2u7yM3rE_M0Nq4b1d1A9OjDSZmPXTMOnHUwCT0JFZJ3Q5UPzSNeuy4GzM_i3Hgub6dggPztrbdBFhg1VbrbvrAStuX8XCi6O2kJ3XZqcwuwn3YVMbP2r3m7mxl5bsReNdDjnf5bjQXlf3sdei8JlGL6yeL5TuPHOyvYPK6f1n2n5c7MJqtWuBhPpsPtUEn0UJsJr2WpSXN5SgaqWMLexPt0QGxE2zjydbv7puz3Ww48qeWl0VGBEnl7lXPelqM6EmgC3RQWjaYKGBp04lF2SSDuYrpGgNFuqaNhDWiq2uTvApcx9L1rsV3ul2N583r2tYOk53QZWvEDwTuHwBMT2Y" }, "search_information": { "ai_overview_state": "Fully empty" }, "error": "Google hasn't returned any results for this query." }, "note": "expansion of the page_token saved in the 2026-09-18 TC2 record" }
- query_text
- If I quit Netflix, do I lose the shares I was granted while I was working there?
- timestamp_utc
- 2026-09-18T09:13:56Z
- raw_response › ai_overview
- { "page_token": "QFM4dHiczVXbcqpIFP2A-Y7xxZSAaI6mikp1t4hoNMF4iXmxuENQWmkIIW_zL_M2fzRfMw2iQoLJnPMwNbsqabpZ7t63xfrjz43re3__9pcThjtywzBxHDdsjO2N2dDxllFJ4uuMhTck0J3b5FXgazoR2JrpChxAoRqMBsiQgvYEPyThdRABuZb9RFhrKjFfqMe3F8KsGfrP-53v0U2DNEy_Qd4Sn7elWDZwo8k2-cY9o24p … [1,391 characters in the file]", "serpapi_link": "https://serpapi.com/search.json?engine=google_ai_overview&page_token=QFM4dHiczVXbcqpIFP2A-Y7xxZSAaI6mikp1t4hoNMF4iXmxuENQWmkIIW_zL_M2fzRfMw2iQoLJnPMwNbsqabpZ7t63xfrjz43re3__9pcThjtywzBxHDdsjO2N2dDxllFJ4uuMhTck0J3b5FXgazoR2JrpChxAoRqMBsiQgvY … [1,460 characters in the file]" }
- raw_response › search_information
- { "query_displayed": "If I quit Netflix, do I lose the shares I was granted while I was working there?", "total_results": 134, "time_taken_displayed": 0.3, "organic_results_state": "Results for exact spelling" }
- raw_response › search_parameters
- { "engine": "google", "q": "If I quit Netflix, do I lose the shares I was granted while I was working there?", "google_domain": "google.com", "device": "desktop" }
An answering page was required in the result list, and none appeared. The AI Overview's content from 18 September 2026 is unknown, but the outcome is not inconclusive: the search completed normally, and an overview is a written answer rather than a retrieved page.
Exact-spelling search. For the Netflix question, SerpAPI passed the wording to Google unchanged. Google used an exact-spelling match and reported 134 total results.
Later AI Overview request. The search supplied a Google AI Overview link without answer text. A later request on 3 October 2026 returned an empty overview.
Configuration and setup
How this tool was set up for the run and what the test needed in place. Each row is a fact from the run's records; a fact the records do not hold is left out, not guessed.
How this scenario is graded
How we decide Pass, Fail and Not gradable. The same rules apply to every tool tested on this scenario.
Each test case gets one result per tool: Pass, Fail or Not gradable. A test case we haven't run yet shows Untested. There are no partial results.
- Pass: the tool did everything the test expected, and nothing it said contradicts the correct answer.
- Fail: at least one expected behaviour clearly didn't happen; the test case's result says which one and quotes the tool.
- Not gradable: our evidence couldn't settle the outcome (for example a record we needed is missing). It is never counted as a fail, and the test case's result says what's missing.
Where this sits in the benchmark
This page is one cell of a larger study: one tool, one scenario. Only this benchmark's frame appears here.
| Level | Name | Scope |
|---|---|---|
| Benchmark | Web Search for AI → | 10 scenarios · 12 tools |
| Capability | Web Retrieval → | |
| Scenario | The request uses different wording than the source → | |
| Tool | SerpAPI → |
Global scenario definition → · Global capability definition → · SerpAPI product page →
History of this result
What has happened to this result since it was first published. Runs and grades are never overwritten: a retest or a re-grade publishes a new result and keeps the earlier one readable.
Act on this result
Nothing filed here edits the run or the grade. A challenge opens a review, and a review can produce a new run or a re-grade — which becomes the current result and leaves this one in the history.
You run the same kind of test against your own setup and get the same behaviour.
Agree →Yours behaves differently. Tell us what you got, with a screenshot if you have one.
Disagree →Something here is wrong — a reference value, a transcription, a grade.
Report an issue →Tell us what changed and we schedule a rerun of the failing test case. The old result stays as history.
Vendor notice →The same record is available as structured data through the AI Demos MCP server, with the counts, the coverage and every per-test-case reason carried as fields.
Verify the proof files
These files support this result. Open a file to inspect the original evidence.
File fingerprints (SHA-256)
A fingerprint identifies the exact file used for this result.