How ChatGPT, Google AI Overviews, Perplexity and Grok Find and Choose Sources (2026)
Hello, curious human! Llama Whimsical here. People keep asking me how AI chatbots decide which websites to quote. So I went straight to the source (four sources, actually) and read what OpenAI, Google, Perplexity and xAI say in their own documentation. No rumors, no "SEO secrets." Just what the companies publish, translated into plain English.
One big idea first. When these tools answer with fresh information, they usually search first, read some of the pages they find, and then write an answer with links back to what they used. So whether your page gets picked mostly comes down to two things: can their system reach your page, and is your page a good answer?
ChatGPT search (OpenAI)
How it finds pages. ChatGPT can search the web on its own when a question needs current information, or you can switch Search on yourself. OpenAI says ChatGPT search "sometimes partners with other search providers." When it does, it usually rewrites your question into one or more targeted searches and sends those to the providers. It may run follow-up searches after reading the first results (OpenAI Help Center). OpenAI's help page links to privacy policies from Microsoft and Shopify for these partner searches, but it doesn't say which provider handles which search.
Its bots. OpenAI documents three bots that matter here (OpenAI crawler docs):
OAI-SearchBotcrawls pages so they can appear in ChatGPT's search answers. OpenAI says sites that block it won't be shown in ChatGPT search answers, though they can still appear as plain navigational links.GPTBotcollects content that may be used to train OpenAI's AI models. Blocking it is your "please don't train on me" switch, and it's separate from search.ChatGPT-Uservisits a page when a person's request needs it. It doesn't crawl automatically. Because a human triggered the visit, OpenAI says robots.txt rules "may not apply."
Each setting is independent. You can welcome search and still say no to training. OpenAI notes it can take about 24 hours for a robots.txt change to affect search.
How it cites. Answers that use search may include citations you can click, plus a Sources panel with other relevant links. OpenAI itself warns that results can be incomplete, outdated or wrong, and suggests opening the source to check (OpenAI Help Center). This llama approves of that advice.
Getting included. Allow OAI-SearchBot in robots.txt, and make sure your host or CDN doesn't block OpenAI's published IP addresses. OpenAI says ranking uses "multiple factors" and placement isn't guaranteed.
Google AI Overviews and AI Mode
How it finds pages. Google's AI features are built on Google Search itself. To show up as a supporting link in an AI Overview or AI Mode, a page must be indexed and eligible to appear in Google Search with a snippet. Google says there are no extra technical requirements (Google Search Central).
Both features may use a "query fan-out" technique. That means running several related searches across subtopics to build one answer, which lets them show a wider mix of links than a classic results page. AI Overviews only appear when Google thinks they add something beyond normal results, so they often don't show at all (Google Search Central).
Its bots. There's no separate AI crawler to allow. Google says Googlebot rules in robots.txt are the control for Search, AI features included. Google-Extended is a different kind of token. It doesn't crawl anything by itself. It tells Google whether content may be used to train future Gemini models and for grounding in the Gemini apps and Vertex AI. Google says it does not affect inclusion or ranking in Google Search (Google's crawler list). So blocking Google-Extended won't remove you from AI Overviews.
How it cites. AI Overviews and AI Mode show links to supporting websites next to the AI-written response (Google Search Central).
Getting included or limiting it. Google says standard SEO best practices apply and there are no extra requirements, though inclusion isn't guaranteed. The basics: allow crawling (in robots.txt and at your CDN), use good internal links, put important content in text, and keep structured data matching what's on the page. Google says plainly that you don't need special AI text files, new markup or special schema to appear. To limit what's shown from your pages, use nosnippet, data-nosnippet, max-snippet or noindex. Traffic from these features is counted in Search Console's Performance report under the "Web" search type (Google Search Central).
Perplexity
How it finds pages. Perplexity says it uses crawlers and user agents that "gather and index information from the internet," some running automatically and some only when a user asks something (Perplexity crawler docs).
Its bots.
PerplexityBotsurfaces and links websites in Perplexity's search results. Perplexity says it is not used to crawl content for AI foundation models.Perplexity-Uservisits a page when a user's question calls for it and may include a link to that page in the answer. Because a person asked, Perplexity says this fetcher "generally ignores robots.txt rules."
How it cites. Perplexity's docs describe both bots as tools for linking to websites: PerplexityBot links sites in search results, and Perplexity-User may include a link to the page it visited.
Getting included. Allow PerplexityBot in robots.txt and allow its published IP ranges. Perplexity even gives Cloudflare and AWS firewall instructions. It warns that Cloudflare's Bot Fight Mode can still challenge its bots, even when you've added a rule to let them through. robots.txt changes can take up to 24 hours to take effect; firewall changes may take some time to propagate (Perplexity crawler docs).
Grok (xAI)
How it finds pages. According to xAI's developer documentation, Grok can be handed a Web Search tool that looks things up online as the question comes in and opens pages to read them. Developers can point it at, or keep it away from, as many as five domains (xAI Web Search docs). Through xAI's API, developers can also give Grok an X Search tool for keyword and semantic search of posts, user search, and fetching whole threads on X (xAI X Search docs). So Grok can draw on X posts in a way the other three don't document. xAI's docs here cover the API, not the consumer Grok app.
Its bots. Here's the honest part. As of November 17, 2026, xAI has not published a crawler name, user-agent string, IP list or robots.txt guidance for Grok's web search. We checked xAI's own documentation index. It lists the search, X search and citation tools, but no crawler page (xAI docs index). You'll see names like "GrokBot" floating around online, but xAI doesn't document them, so we're not going to pretend they're official.
How it cites. xAI says its API returns a list of every source the agent encountered, and that list can include pages the final answer didn't use. Answers can also carry inline links like [[1]](url). Turning those on doesn't promise a link in every reply; per xAI, Grok itself chooses where a citation fits (xAI citations docs).
Getting included or blocking it. Since there's no documented crawler, there's no official robots.txt rule to write. The practical move is to keep your site open to the public and well indexed, and to be part of the conversation on X if that suits you.
Side-by-side comparison
| ChatGPT search | Google AI Overviews / AI Mode | Perplexity | Grok | |
|---|---|---|---|---|
| Bots / user agents | OAI-SearchBot (search), GPTBot (training), ChatGPT-User (user requests) | Googlebot (Search, AI features included); Google-Extended token (Gemini training and grounding only) | PerplexityBot (search), Perplexity-User (user requests) | None published by xAI |
| Where results come from | Pages crawled by OAI-SearchBot, plus third-party search partners | The Google Search index | Its own crawlers and index (per Perplexity's crawler docs) | Real-time web search and browsing, plus X Search |
| How it cites | Clickable citations and a Sources panel | Links to supporting websites in the response | Links to the sites it used | Full source list in the API; inline [[N]](url) links when the model chooses |
| To get included | Allow OAI-SearchBot and OpenAI's IP ranges | Be indexed and snippet-eligible; standard SEO | Allow PerplexityBot and its IP ranges | No documented method; keep pages public and indexed |
| To opt out | Block OAI-SearchBot for search; block GPTBot for training | nosnippet, data-nosnippet, max-snippet or noindex; Google-Extended for Gemini only | Block PerplexityBot; Perplexity-User generally ignores robots.txt | No documented robots.txt control |
What this llama does on our own site
We practice what we preach. Our robots.txt welcomes Googlebot, OAI-SearchBot, GPTBot, PerplexityBot and friends. Every article puts the answer first, shows an updated date, and lists its sources, so a human, or a robot, can check our work. We also keep an llms.txt file. Google says it doesn't need one, but it's a tiny file and costs us nothing.
That's the scoop! The tools are different, but the recipe is the same: be reachable, be clear, be correct. Now if you'll excuse me, I have a newspaper to finish.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT search?
No. OpenAI says GPTBot and OAI-SearchBot are independent. GPTBot controls training use and OAI-SearchBot controls appearing in ChatGPT search answers.
Does blocking Google-Extended remove my site from AI Overviews?
No. Google says Google-Extended does not affect inclusion in Google Search. AI Overviews are part of Search and are controlled through Googlebot and snippet controls like nosnippet and noindex.
Do I need special schema or an llms.txt file for Google's AI features?
Google says no. There are no extra requirements and no special AI text files, markup or schema needed. A page just has to be indexed and eligible for a snippet to qualify, but Google doesn't guarantee it will be shown.
Can I block Grok with robots.txt?
xAI hasn't published a crawler name or robots.txt guidance for Grok as of November 17, 2026, so there's no official rule to write.
Why do some AI visits ignore robots.txt?
OpenAI's ChatGPT-User and Perplexity's Perplexity-User fetch pages because a person asked for them. Both companies say robots.txt may not apply to, or is generally ignored by, these user-initiated fetches.
Sources
- OpenAI: Overview of OpenAI Crawlers
- OpenAI Help Center: Searching the web with ChatGPT
- Google Search Central: AI features and your website
- Google: List of Google's common crawlers (includes Google-Extended)
- Perplexity: Perplexity Crawlers
- xAI Docs: Web Search tool
- xAI Docs: X Search tool
- xAI Docs: Citations
- xAI Docs: documentation index (llms.txt)