How ChatGPT and Perplexity Choose Their Sources (and How to Become One)
ChatGPT chooses its sources in two stages. First it rewrites your question into one or more search queries and sends them to a web index, using its own crawler (OAI-SearchBot) together with search partners OpenAI names, including Microsoft. Then the model reads the returned pages and decides which passages to use and cite, applying what Ahrefs calls its own selection layer: only about 8 to 10 percent of the URLs ChatGPT cites rank in Google’s top ten for the same query. Perplexity works the same way but stays closer to conventional search, with roughly 29 to 65 percent of its citations overlapping Google’s top ten depending on query type. Neither company publishes a ranking formula, but the pattern across published citation studies is consistent: both engines favor pages they can retrieve, passages that answer a sub-question directly, and sources that corroborate each other. To become a source you need to be crawlable by the right bot, rank for the sub-queries the engine generates, write answers that can be lifted whole, and be confirmed by independent sites the engines already trust, especially Reddit, Wikipedia, LinkedIn and recognized publications.
Key takeaways
- ChatGPT search eligibility is controlled by OAI-SearchBot, not GPTBot. Blocking GPTBot (training) does not remove you from ChatGPT search; blocking OAI-SearchBot does.
- Perplexity uses PerplexityBot for its index, and its user-triggered fetcher (Perplexity-User) generally ignores robots.txt. Both engines say robots.txt changes take about 24 hours.
- Ahrefs found that across AI assistants only 12 percent of cited URLs rank in Google’s top ten for the prompt, and about 80 percent do not rank anywhere in Google for it. Retrieval is wider than page one.
- Semrush’s 13-week study of 230,000 prompts found Reddit and Wikipedia were ChatGPT’s two most-cited domains, and LinkedIn was in the top five on ChatGPT, Perplexity and Google AI Mode.
- For “best” and “should I use” questions, the engines cite third parties that make the recommendation, not the brand itself, which is why press and independent profiles decide whether you are named.
How does ChatGPT choose its sources?
ChatGPT does not have a fixed list of trusted sites. For most factual and commercial questions it runs a retrieval step and then a selection step, and the two have different rules.
Step one: retrieval
According to OpenAI’s ChatGPT search help page, the model rewrites your question into one or more targeted queries, sends them to search partners, reviews the results, and may send follow-up queries. OpenAI says it sometimes partners with other search providers and names Microsoft and Shopify, and that results are ranked using multiple factors with no guaranteed placement. The launch announcement from October 2024 adds that the system draws on outside search providers and on content that partners supply directly, with named publisher partners including the Associated Press, Reuters, the Financial Times, Condé Nast, Hearst, News Corp, Time, The Atlantic and Vox Media.
The practical rule is that a page has to be in the index ChatGPT searches and has to be retrievable for one of the rewritten queries. OpenAI’s help page says sites become eligible by allowing OAI-SearchBot to crawl them and by permitting traffic from OpenAI’s published searchbot IP ranges. Eligibility does not guarantee appearance.
Step two: selection
Once pages are retrieved, the model decides which to read closely and which to cite. This is where ChatGPT departs from a search engine. In a September 2025 study of 3,311 head-term keywords, Ahrefs found that only about 10 percent of the URLs ChatGPT cited were in Google’s top ten for the same keyword, while domain-level overlap was 31.8 percent. In other words, ChatGPT tends to cite the same domains Google trusts but different pages on them, and Ahrefs concluded it “applies its own selection layer that filters and reshuffles which links appear.” A separate Ahrefs analysis of 15,000 long-tail prompts, published in August 2025, put the figure lower still: 8 percent of ChatGPT’s in-text citations ranked in Google’s top ten, and about 80 percent of AI citations across assistants did not rank anywhere in Google for the prompt.
What survives the selection layer is a passage, not a page. The model is looking for a sentence or paragraph that resolves one of its sub-queries cleanly and that it can attribute without hedging. Pages that state the answer up front, with a number, a date and a named source, are cited; pages that bury it are read and discarded. The broader framework for why this is so is in our pillar on what generative engine optimization is.
How does Perplexity choose its sources?
Perplexity describes itself as an answer engine and its process is closer to classic search than ChatGPT’s. Its help center describes three steps: interpret the question, search the web in real time, and summarize the results into a single answer with numbered citations that link back to the original pages. Pro Search runs research across multiple sources and Deep Research runs dozens of searches and reads hundreds of pages before writing a report. Perplexity says it gathers from “authoritative sources like articles, websites, and journals” but does not publish how those are ranked.
The index is built by PerplexityBot. Perplexity’s crawler documentation says PerplexityBot surfaces and links websites in search results and is not used to crawl content for training foundation models; to appear, a site should allow it in robots.txt and permit its published IP ranges. A second agent, Perplexity-User, fetches a page when a user asks a question that needs it, and the documentation states that “since a user requested the fetch, this fetcher generally ignores robots.txt rules.” Changes to robots.txt take up to 24 hours.
In practice Perplexity’s citations track Google more closely than ChatGPT’s. Ahrefs measured 28.6 percent overlap with Google’s top ten on long-tail prompts and about 65 percent on head terms, the highest of any assistant tested. If you rank on page one for a commercial query, you are a realistic Perplexity source; on ChatGPT, ranking is necessary but much less sufficient.
| Engine | Index and crawler | Overlap of cited URLs with Google top 10 (Ahrefs) | Selection behavior | How to control appearance |
|---|---|---|---|---|
| ChatGPT search | OAI-SearchBot plus search partners (Microsoft named) | About 8% on long-tail prompts, about 10% on head terms; 31.8% domain overlap | Strong model-side filtering; favors clean, extractable passages and domains it already trusts | Allow or disallow OAI-SearchBot; allow OpenAI’s searchbot IP ranges; ~24 hours to take effect |
| Perplexity | PerplexityBot (own index) plus real-time fetching | 28.6% on long-tail prompts, about 65% on head terms | Tracks conventional rankings closely; numbered citations per claim | Allow or disallow PerplexityBot; Perplexity-User generally ignores robots.txt; ~24 hours |
| Google AI Mode and AI Overviews | Google Search index via Googlebot | Not measured in the Ahrefs studies above; built on Search rankings with query fan-out | Pulls passages for sub-queries; wider set of links than classic results | Standard indexing and snippet controls; see our AI Overviews guide |
Google’s features behave differently enough to need their own guide: how to rank in Google AI Overviews and AI Mode.
What do ChatGPT and Perplexity actually cite?
The largest public dataset on this question is Semrush’s study of the most-cited domains in AI, published in November 2025. It tracked about 230,000 prompts weekly across ChatGPT search, Google AI Mode and Perplexity from July 14 to October 12, 2025, covering more than 100 million citations. The findings:
- ChatGPT: Reddit and Wikipedia were the two most-cited domains, at roughly five times the next tier until mid-September. Reddit appeared in close to 60 percent of ChatGPT responses in early August and fell to around 10 percent by mid-September; Wikipedia fell from about 55 percent of responses to under 20 percent. Medium, Forbes and LinkedIn rounded out the top group, and PRNewswire, Forbes and Medium gained share after the September shift.
- Perplexity: Top sources were Reddit, LinkedIn, NIH, Microsoft and Google. Reddit’s citations were more stable here than on ChatGPT, declining only slightly in September.
- Google AI Mode: LinkedIn, YouTube, Reddit, Google and Google’s own blog led, and LinkedIn appeared in nearly 15 percent of responses. YouTube was one of the biggest gainers after September.
- Across all three: LinkedIn was in the top five everywhere, which is unusual for a platform most brands treat as a résumé site rather than a publishing channel.
Three lessons follow. The engines lean heavily on a few community and reference platforms where real people describe real experiences, so a brand absent from Reddit, LinkedIn and reference entries is absent from much of the raw material. Recognized publications and newswire content are a durable part of the mix, and their share rose when community platforms fell. And the mix changes fast: ChatGPT’s Reddit share fell by roughly five-sixths in six weeks, so a source strategy built on one platform is fragile. The comparison between being mentioned and being linked is drawn out in brand mentions vs backlinks for AI answers.
Which crawlers matter, and what should your robots.txt allow?
Each AI company runs more than one bot, and the one that governs training is not the one that governs search. Blocking the wrong agent either removes you from answers you wanted to appear in or fails to protect content you wanted to protect.
| Company | User agent | Purpose | Effect of blocking in robots.txt |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Indexes sites for ChatGPT search | Site will not appear in ChatGPT search answers (may still appear as a navigational link) |
| OpenAI | GPTBot | Crawls content that may be used to train models | Signals content should not be used for training; no effect on search appearance |
| OpenAI | ChatGPT-User | Fetches a page when a user’s request requires it | Not used for automatic crawling; robots.txt may not apply because a user initiated the action |
| Perplexity | PerplexityBot | Indexes sites for Perplexity search results | Site will not be surfaced from the index |
| Perplexity | Perplexity-User | Fetches a page in response to a user question | Generally ignores robots.txt because the user requested it |
| Anthropic | Claude-SearchBot, Claude-User, ClaudeBot | Search indexing, user-requested fetches, and training respectively | Blocking Claude-SearchBot or Claude-User may reduce visibility in Claude’s web answers; blocking ClaudeBot opts out of training |
The OpenAI entries come from its bot documentation, which also notes that if a site allows both OAI-SearchBot and GPTBot, one crawl’s results may be used for both purposes, and that robots.txt changes for search can take about 24 hours. The Anthropic entries come from its crawler support article. Two further points matter. Many CDNs and firewalls block unfamiliar bots by default, so a site can have a permissive robots.txt and still be invisible; both companies publish IP ranges and ask that they be allowed at the network layer. And blocking training bots does not remove what the models already learned, so a brand with a bad description in the training data still needs search-time sources to correct it. How to find out what the models currently say is covered in what ChatGPT says about your company.
Why freshness and specificity win the tie
Both engines search live, which changes what “authoritative” means. A page from 2019 with a strong backlink profile is a strong Google result; to an engine answering “what does X cost in 2026,” it is a stale candidate. OpenAI’s help page says ChatGPT may issue follow-up queries after reviewing initial results, and Perplexity describes real-time search; in both cases a visibly current page with a dated update and a precise figure is the safer thing for the model to quote, because the model is accountable for the citation.
Specificity works the same way. A sub-query like “typical PR retainer length” is answered by a sentence that says “three to twelve months,” not by a paragraph that says it depends. The original generative engine optimization research found that adding statistics, quotations and source citations to a page produced the largest visibility gains in generated answers, while keyword stuffing reduced visibility below the unoptimized baseline. The engines are not rewarding density; they are rewarding sentences that can be lifted and defended.
One more pattern is visible in the Semrush data: newswire and press content gained ChatGPT citation share in autumn 2025 when community platforms lost it. Dated, bylined, factual articles in recognized outlets are the kind of source a model can cite without risk.
Why third-party corroboration decides recommendations
There is a difference between being cited for information and being named as a recommendation. If a user asks how PR retainers work, ChatGPT will cite whoever explains it best, and that can be the agency’s own blog. If the user asks which PR agency to hire, the model is making a judgment, and it will not rest that judgment on the agency’s own claims. It looks for sources that make the recommendation independently: review platforms, comparison articles, industry publications, Reddit threads where practitioners name names, LinkedIn posts from clients, and news features. The domains that dominate the citation studies above are dominant precisely because they are where third parties talk about brands.
This is the mechanism behind a pattern many businesses see: strong rankings, good content, and no presence in AI recommendations. The content is retrieved, then the model checks the recommendation against the rest of the web and finds nothing. Rankings alone do not close this gap. The way coverage converts into being named is laid out in how press coverage gets your brand recommended by ChatGPT.
Corroboration also has to be consistent. A model reconciling five sources that describe a company five different ways will either hedge or omit it. The same founding year, services, leadership and positioning should appear on the site, on LinkedIn, in Crunchbase and Wikidata where eligible, in press coverage and in review profiles. Consistency is what lets the engine resolve the brand to one entity and speak about it with confidence.
A checklist for becoming a source
The following steps are ordered by dependency: each one is wasted if the one before it is not done.
- Confirm you are crawlable by the search bots. Check robots.txt for OAI-SearchBot and PerplexityBot, then check your CDN or firewall logs for blocked requests from OpenAI’s and Perplexity’s published IP ranges. Fix the network layer, not just the text file.
- Rank for the sub-queries, not only the head term. List the ten to twenty questions a buyer asks on the way to your category, and make sure a page on your site is a credible answer to each. Perplexity in particular rewards conventional rankings.
- Lead every section with the answer. One to three sentences that fully resolve the heading’s question, with a number, a date and a source where one exists. The rest of the section is support.
- Put definitions, comparisons and processes in liftable form. A definition in a single sentence, a comparison in a table, a process in a numbered list. These are the shapes the models extract most accurately.
- Date and sign your pages. Visible update dates, named authors and Organization markup give live-searching engines the freshness and trust signals they weigh.
- Be present where the engines already look. Accurate, complete LinkedIn company and founder profiles; participation in the Reddit communities where your category is discussed, under Reddit’s rules and without astroturfing; a Wikidata entry if you qualify; and current listings on the review and directory sites the engines cite in your vertical.
- Earn independent coverage that makes the recommendation. Interviews, expert quotes and features in recognized outlets are the sources a model can cite when it names you as an option. Guaranteed editorial placements are one route when earned coverage is slow.
- Keep every description consistent. Audit your entity facts across all of the above twice a year and after any rebrand, funding round or leadership change.
- Measure with a fixed prompt set. Run the same 20 to 50 prompts monthly on each engine and log mentions, citations and accuracy. The method and the tools are covered in AI visibility tools and metrics.
What is not on the list: a special file for AI crawlers. The llms.txt proposal is discussed in llms.txt explained; no engine has confirmed using it to select sources, and server-log studies show the search bots rarely request it.
Where Viral Spike fits
Viral Spike Marketing’s AI search visibility service starts with the measurement step above: a fixed set of category and brand prompts run across ChatGPT, Perplexity, Gemini, Claude and Google’s AI features, with every cited domain logged, so you know which sources the engines are using for your category and whether you are among them. From there the work splits into the technical layer (crawler access, extractable pages, entity markup), the presence layer (profiles and platforms the engines cite), and the corroboration layer, which is built on guaranteed editorial placements in recognized outlets: you approve the draft, the article is published and indexed on Google, and the engagement is refunded in full if it never runs. Our brand authority program combines all three layers for brands that want one accountable plan. To see the baseline for your category first, contact us.
Check the crawler, then the passage, then the corroboration
Start with three ten-minute checks this week. Read your robots.txt and your firewall rules for OAI-SearchBot and PerplexityBot. Ask ChatGPT and Perplexity the five questions a prospect would ask before hiring someone like you, and note which domains they cite. Then search Reddit, LinkedIn and Google News for your brand and see what a model would find if it went looking for a second opinion. The gaps in those three checks are the plan: open the crawler path, rewrite the pages so the answer is the first sentence, and build the independent coverage that lets an engine recommend you without taking your word for it.
Frequently asked questions
How does ChatGPT decide which websites to cite?
ChatGPT rewrites your question into search queries, retrieves pages through OAI-SearchBot’s index and search partners such as Microsoft, then applies its own selection layer to decide which passages to quote and cite. Ahrefs found only about 8 to 10 percent of cited URLs rank in Google’s top ten for the same query, though domain overlap is about 32 percent. Clean, specific, dated passages on domains the model already trusts are cited most.
Does blocking GPTBot remove my site from ChatGPT search?
No. OpenAI’s bot documentation states that GPTBot governs training data and that blocking it does not affect search appearance. ChatGPT search eligibility is controlled by OAI-SearchBot; disallowing that agent in robots.txt removes your site from ChatGPT search answers, though it may still appear as a navigational link. Changes take about 24 hours. Your host or CDN must also allow OpenAI’s published IP ranges.
How does Perplexity pick its sources?
Perplexity interprets the question, searches the web in real time through its PerplexityBot index and user-triggered fetches, then summarizes results with numbered citations linking to each source. It does not publish ranking criteria, but Ahrefs measured 28.6 percent overlap with Google’s top ten on long-tail prompts and about 65 percent on head terms, so conventional rankings matter more for Perplexity than for ChatGPT.
Which websites do AI engines cite most often?
Semrush’s study of 230,000 prompts from July to October 2025 found Reddit and Wikipedia were ChatGPT’s two most-cited domains, with Medium, Forbes and LinkedIn following. Perplexity’s top sources were Reddit, LinkedIn, NIH, Microsoft and Google. Google AI Mode leaned on LinkedIn, YouTube and Reddit. LinkedIn was in the top five on all three platforms, and shares shifted sharply within weeks.
Why does ChatGPT recommend competitors instead of my brand?
Usually because independent sources recommend them and not you. For informational questions the model cites the clearest explanation, which can be your own page. For recommendation questions it checks third-party sources such as review platforms, Reddit, LinkedIn and publications, and names brands that appear there consistently. The fix is coverage and consistent entity data across those sources, not more on-site content.
Does adding an llms.txt file help ChatGPT or Perplexity cite me?
There is no evidence that it does. Neither OpenAI nor Perplexity has said their search systems use llms.txt, and a 2026 Ahrefs analysis of 137,000 domains found that 97 percent of published llms.txt files received no requests at all in a month. Time is better spent on crawler access, answer-first pages and independent coverage.
Sources
Primary sources referenced in this article, in order of first use.
- ChatGPT search help page (help.openai.com)
- launch announcement from October 2024 (openai.com)
- Ahrefs (ahrefs.com)
- 8 percent of ChatGPT's in-text citations (ahrefs.com)
- help center (perplexity.ai)
- crawler documentation (docs.perplexity.ai)
- the most-cited domains in AI (semrush.com)
- bot documentation (developers.openai.com)
- crawler support article (support.claude.com)
More in AI Search
Related services