Can Exa search only the allowed websites?
Exa met the site-only request. It returned docs.github.com results, including the Copilot page at rank 2, and no GitHub blog result; a rank-5 preview-policy page mentions other dates but not the Copilot launch year.
1 of 1 test case passed
Every test case in this scenario has a result.
The test case
Each test case is judged on its own: Pass, Fail, Not gradable, or Untested. The scenario result above is based on this test case.
Search only these websitesPassEvidence
Input · Search limited to docs.github.com (includeDomains filter): “In what year did GitHub first launch a technical preview of GitHub Copilot?”
{ "requestId": "abbbd42db87964c784779f2fb3ca271f", "resolvedSearchType": "", "results": [ { "id": "https://docs.github.com/en/get-started/using-github/exploring-early-access-releases-with-feature-preview", "title": "", "url": "https://docs.github.com/en/get-started/using-github/exploring-early-access-releases-with-feature-preview", "text": "# Exploring early access releases with feature preview\n\nYou can use feature preview to see products or features that are available in public preview and to enable or disable each feature for your personal account.\n\n## GitHub's release cycle … [1,827 characters in the file]" }, { "id": "https://docs.github.com/en/copilot/get-started/what-is-github-copilot", "title": "", "url": "https://docs.github.com/en/copilot/get-started/what-is-github-copilot", "text": "# What is GitHub Copilot?\n\nLearn what Copilot is and what you can do with it.\n\nGitHub Copilot is an AI coding assistant that helps you write code faster and with less effort. Then, you can focus more energy on problem solving and collaborat … [3,388 characters in the file]" }, { "id": "https://docs.github.com/en/copilot/concepts/learning-about-new-features-and-models", "title": "", "url": "https://docs.github.com/en/copilot/concepts/learning-about-new-features-and-models", "text": "# Learning about new features and models\n\nStay informed about GitHub Copilot features and models, and make confident decisions about enabling them for your enterprise.\n\nGitHub Copilot is evolving rapidly, with new features and models releas … [5,603 characters in the file]" }, { "id": "https://docs.github.com/en/copilot/concepts/preparing-for-new-features-and-models", "title": "", "url": "https://docs.github.com/en/copilot/concepts/preparing-for-new-features-and-models", "text": "# Learning about new features and models\n\nStay informed about GitHub Copilot features and models, and make confident decisions about enabling them for your enterprise.\n\nGitHub Copilot is evolving rapidly, with new features and models releas … [6,256 characters in the file]" }, { "id": "https://docs.github.com/en/site-policy/github-terms/github-dpa-previews", "title": "GitHub DPA-Covered Previews - GitHub Docs", "url": "https://docs.github.com/en/site-policy/github-terms/github-dpa-previews", "publishedDate": "2025-10-27T00:00:00.000Z", "text": "# GitHub DPA-Covered Previews\n\n## GitHub DPA-Covered Previews\n\nThe pre-release products and features listed below use data handling that is the same as when the software becomes generally available. Under GitHub’s Pre-Release License Terms, … [780 characters in the file]" }, { "id": "https://docs.github.com/en/copilot", "title": "GitHub Copilot documentation - GitHub Docs", "url": "https://docs.github.com/en/copilot", "text": "# GitHub Copilot documentation\n\nYou can use GitHub Copilot to enhance your productivity and assist as you work on code.\n\n## Links\n\n### Getting started\n\n- What is GitHub Copilot? Learn what Copilot is and what you can do with it.\n- Quickstar … [65,855 characters in the file]" }, { "id": "https://docs.github.com/en/copilot/get-started", "title": "Get started with GitHub Copilot", "url": "https://docs.github.com/en/copilot/get-started", "text": "# Get started with GitHub Copilot\n\nLearn how to sign up for and use GitHub Copilot.\n\n## Links\n\n* Quickstart for GitHub Copilot\n\n Quickly learn how to use GitHub Copilot.\n\n* What is GitHub Copilot?\n\n Learn what Copilot is and what you can do … [649 characters in the file]" }, { "id": "https://docs.github.com/en/copilot/get-started/features", "title": "", "url": "https://docs.github.com/en/copilot/get-started/features", "text": "# GitHub Copilot features\n\nGitHub Copilot offers a suite of features for users and administrators.\n\n## Assistive features\n\nThese tools are used synchronously, providing advice or suggestions as people work on a task.\n\n### Copilot Chat\n\nA ch … [6,468 characters in the file]" }, { "id": "https://docs.github.com/en/copilot/concepts", "title": "Concepts for GitHub Copilot", "url": "https://docs.github.com/en/copilot/concepts", "text": "# Concepts for GitHub Copilot\n\nLearn the core concepts that you'll need to understand GitHub Copilot.\n\n## Links\n\n* Completions for GitHub Copilot\n\n Learn how Copilot can provide inline suggestions.\n\n* About GitHub Copilot Chat\n\n Learn how y … [2,393 characters in the file]" }, { "id": "https://docs.github.com/en/copilot/concepts/agents/copilot-cli/about-copilot-cli", "title": "About GitHub Copilot CLI", "url": "https://docs.github.com/en/copilot/concepts/agents/copilot-cli/about-copilot-cli", "text": "# About GitHub Copilot CLI\n\nFind out about using Copilot from the command line.\n\n## Introduction\n\nThe command-line interface (CLI) for GitHub Copilot allows you to use Copilot directly from your terminal. You can use it to answer questions, … [22,158 characters in the file]", "image": "https://docs.github.com/assets/images/help/copilot/copilot-cli-welcome.png" } ], "searchTime": 4581.9, "costDollars": { "total": 0.007, "search": { "neural": 0.007 } } }
Search response: permitted documentation pages, including the Copilot introduction at rank 2
Copied from Output file (JSON), raw_response · an excerpt
Open the whole JSON file ↗| ✓ | Expected: Exclude GitHub's blog announcement and return the relevant Copilot introduction page from the permitted GitHub documentation site.Found: All ten results came from GitHub Docs, including the required Copilot introduction at rank 2. GitHub's blog announcement did not appear. |
- answer_text
- null
- status
- ok
Exa retained the required relevant page while excluding the answer-bearing blog, satisfying both parts of the restriction. The result covers search with Exa's native domain filter, not a generated answer.
Configuration and setup
How this tool was set up for the run and what the test needed in place. Each row is a fact from the run's records; a fact the records do not hold is left out, not guessed.
How this scenario is graded
How we decide Pass, Fail and Not gradable. The same rules apply to every tool tested on this scenario.
Each test case gets one result per tool: Pass, Fail or Not gradable. A test case we haven't run yet shows Untested. There are no partial results.
- Pass: the tool did everything the test expected, and nothing it said contradicts the correct answer.
- Fail: at least one expected behaviour clearly didn't happen; the test case's result says which one and quotes the tool.
- Not gradable: our evidence couldn't settle the outcome (for example a record we needed is missing). It is never counted as a fail, and the test case's result says what's missing.
Where this sits in the benchmark
This page is one cell of a larger study: one tool, one scenario. Only this benchmark's frame appears here.
| Level | Name | Scope |
|---|---|---|
| Benchmark | Web Search for AI → | 10 scenarios · 12 tools |
| Capability | Web Retrieval → | |
| Scenario | Search only these websites → | |
| Tool | Exa → |
Global scenario definition → · Global capability definition → · Exa product page →
History of this result
What has happened to this result since it was first published. Runs and grades are never overwritten: a retest or a re-grade publishes a new result and keeps the earlier one readable.
Act on this result
Nothing filed here edits the run or the grade. A challenge opens a review, and a review can produce a new run or a re-grade — which becomes the current result and leaves this one in the history.
You run the same kind of test against your own setup and get the same behaviour.
Agree →Yours behaves differently. Tell us what you got, with a screenshot if you have one.
Disagree →Something here is wrong — a reference value, a transcription, a grade.
Report an issue →Tell us what changed and we schedule a rerun of the failing test case. The old result stays as history.
Vendor notice →The same record is available as structured data through the AI Demos MCP server, with the counts, the coverage and every per-test-case reason carried as fields.
Verify the proof files
These files support this result. Open a file to inspect the original evidence.
File fingerprints (SHA-256)
A fingerprint identifies the exact file used for this result.