← All articles
AI Search9 min read

How to Get Cited by ChatGPT, Perplexity and AI Overviews

How answer engines retrieve and quote sources, why crawler access is the hard prerequisite, and what page structure survives being pulled out of context


Getting cited by an answer engine is a two-stage problem, and almost every "GEO checklist" you have read collapses them into one. Stage one is retrieval: a system pulls a handful of candidate documents or passages from an index in response to a query. Stage two is synthesis: a language model reads those candidates, writes an answer, and attaches citations to whichever passages it actually leaned on. You can win stage one and lose stage two — your page gets fetched, the model reads it, and cites the competitor whose second paragraph stated the answer in a single sentence. Optimising for citation means optimising for both, and they reward different things.

Before going further, the honest caveat that most articles on this topic skip: none of the major engines publish their source-selection criteria. OpenAI, Anthropic, Perplexity and Google all document their crawlers and some of their retrieval architecture, but none of them publish a ranking specification for which of the retrieved sources ends up quoted. Everything in this article splits into two buckets: things vendors have documented (crawler names, robots.txt tokens, index dependencies) which are reliable, and inference from observed behaviour (what kinds of passages get quoted) which is not. I have flagged which is which throughout. Treat the second bucket as reasonable bets, not laws.

How retrieval-augmented answers actually get built

A classic search ranking is a total ordering over documents. A retrieval-augmented answer is not. The common shape is:

  1. The user's question is rewritten — often expanded into several sub-queries. "Is Postgres or MySQL better for JSON" might fan out into three or four distinct searches.
  2. Each sub-query hits an index. That index may be the engine's own crawl, a licensed third-party search API, or a hybrid of keyword (BM25-style) and vector similarity search.
  3. Top candidates are fetched or read from cache, then chunked — split into passages of a few hundred tokens.
  4. Chunks are re-ranked, and a subset is stuffed into the model's context window along with the original question.
  5. The model writes an answer and emits citations against the chunks it used.

Three consequences fall directly out of that pipeline, and they are the whole game.

The unit of competition is the passage, not the page. Your 4,000-word pillar article does not enter the context window. Two or three chunks of it do, selected without the surrounding argument, without your H1, and usually without knowing which site they came from. A paragraph that begins "As we saw above, this is why the second approach wins" is worthless out of context. A paragraph that begins "Connection pooling fails under serverless because each function instance opens its own pool" is a self-contained claim that can be quoted verbatim.

Query fan-out means you are competing for questions you never targeted. The sub-queries are generated by a model, not typed by a human, so they are phrased as full questions with explicit entities. This is part of why long-tail, plainly-phrased H2s ("Why do connection pools break in serverless?") tend to earn citations that keyword-stuffed headings do not.

Freshness and crawl recency matter more than in classic search, because several engines fetch live at answer time rather than serving from a months-old index. If your page 404s, redirects through a chain, or takes eight seconds to respond during that live fetch, you are simply absent. Worth auditing your redirect chains with this in mind — a chain that costs Googlebot a little crawl budget can cost an answer engine the citation outright, because the fetch is happening inside a latency budget measured in seconds.

The prerequisite: are you even fetchable?

Everything above is irrelevant if the crawler is blocked. This is the single most common reason a technically excellent site never appears in AI answers, and it is frequently not a deliberate choice — it is a CDN default, a WAF rule, or a robots.txt line someone added in 2023 when blocking AI crawlers was fashionable.

Here are the tokens that matter, and what each actually controls. These are documented by the vendors, so this table is the reliable bucket:

Token Operator What it governs
GPTBot OpenAI Bulk crawling, primarily for model training data
OAI-SearchBot OpenAI Crawling for ChatGPT search results and links
ChatGPT-User OpenAI Live fetch when a user or tool triggers a page visit
ClaudeBot Anthropic Bulk crawling
Claude-SearchBot Anthropic Crawling to support search results in Claude
Claude-User Anthropic Live fetch on a user request
PerplexityBot Perplexity Crawling for the Perplexity index
Perplexity-User Perplexity Live fetch when a user's question triggers a visit
Google-Extended Google Whether content trains and grounds Gemini apps — not whether you appear in Search
Applebot-Extended Apple Whether content trains Apple's generative models
Bingbot Microsoft The Bing index, which backs Copilot answers

The Google-Extended row deserves its own paragraph because it is misunderstood constantly. Google's documentation is explicit that Google-Extended does not affect inclusion or ranking in Google Search. AI Overviews are built on top of Google Search infrastructure and are crawled by Googlebot. So blocking Google-Extended is a decision about training and Gemini grounding; it is not a lever for AI Overviews, and there is no documented lever for opting out of AI Overviews while remaining in Search other than nosnippet, max-snippet, and data-nosnippet, which suppress your text in ordinary result snippets too.

A robots.txt that allows the major AI crawlers

A subtlety that catches people out: under RFC 9309, a crawler obeys exactly one group — the one whose user-agent token matches most specifically. It does not merge your specific group with the * group. So if you write a permissive group for GPTBot and put your real crawl rules under *, GPTBot will ignore all of your real rules, including the ones protecting /admin/. Repeat the shared rules in each group.

# https://example.com/robots.txt

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /*?session=

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /*?session=

Sitemap: https://example.com/sitemap.xml

Grouping several User-agent lines above one block of rules is valid and applies the block to all of them. Allow: / is redundant against an empty ruleset but harmless, and it makes intent legible to the next person who edits the file. Note that Crawl-delay is not part of the standard and is ignored by Google; some other crawlers honour it, which is worth knowing if you are rate-limiting rather than blocking.

Two things robots.txt cannot fix:

  • CDN and WAF rules. Cloudflare began blocking AI crawlers by default for new domains in mid-2025, and most WAFs ship bot-management rulesets that classify these agents as bots regardless of what your robots.txt says. A permissive robots.txt in front of a 403 is theatre. Check your edge configuration, not just the file.
  • Server-side reality. If the crawler gets a 200 with an empty <div id="root">, allowing it changes nothing. Most AI crawlers should be assumed not to execute JavaScript — vendors are vague here, and behaviour has changed over time, but client-rendered content is the single most common cause of "we're allowed but never cited." If you are on a JS framework, this is worth verifying rather than assuming; the failure modes are covered in more detail in Next.js SEO problems.

Verify with curl rather than trusting the config:

curl -sI -A "PerplexityBot" https://example.com/pricing
curl -s  -A "GPTBot" https://example.com/pricing | head -c 2000

If the second command returns a shell of markup with no prose in it, that is your answer.

Writing passages that survive extraction

Now the inference bucket. Nobody at OpenAI or Perplexity has published "we prefer pages that state the claim early." But the mechanism above makes it a well-motivated bet: chunks are scored for relevance to a question, and a chunk whose first sentence answers that question is more likely to score well and more likely to be quotable once the model has it.

Answer first, then qualify. Under a heading phrased as a question, the first sentence should be the answer. Not "There are several factors to consider when choosing between X and Y." Rather: "Use X when your write volume exceeds roughly 10,000 rows per second; below that, Y's operational simplicity wins." Then spend the next three paragraphs on nuance. The nuance is what makes the article good; the first sentence is what makes it citable.

Make paragraphs self-contained. Practical test: copy any single paragraph into a blank document. Does it still assert something specific and comprehensible? Pronouns without antecedents ("this approach", "as mentioned"), and numbers without units or subjects, all fail this test. You do not have to write robotically — you have to resolve references within each paragraph often enough that a random 300-token window is still meaningful.

Front-load entities. Retrieval is partly lexical. If the page is about Postgres connection pooling in AWS Lambda, those exact strings need to appear in the first paragraph and in headings, not just implied by context. This is the one place where old-fashioned keyword discipline genuinely maps onto the new mechanism.

Use headings as an outline of questions. Chunking strategies frequently split on heading boundaries, which means your H2s and H3s are doing double duty as chunk delimiters and as retrievable text. A flat, honest heading hierarchy helps here for structural reasons, not just accessibility ones — see heading structure rules for the specifics of what "flat and honest" means.

Tables and lists punch above their weight for comparison and specification queries, because they encode relationships densely and survive chunking as coherent units. A markdown-clean HTML table of "feature / plan / limit" is far more extractable than the same content narrated across four paragraphs.

Structured data and entity clarity

No engine has stated that schema.org markup is an input to LLM citation selection. What is documented is that Google uses structured data for rich results, and that several answer engines consume publicly available structured data to disambiguate entities. The reasonable position: markup is cheap, it is unambiguous, and it costs you nothing if it turns out to be ignored.

The types with the clearest payoff are the boring ones. Organization with sameAs links to your Wikipedia, Crunchbase, GitHub and LinkedIn profiles is the closest thing to telling a retrieval system "this string refers to this entity." Article with a real author (a Person with their own sameAs links, not a byline string) and an accurate dateModified. FAQPage only where the page genuinely is a list of questions. Product with offers if you sell something and want price accuracy in answers about your pricing.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Example Corp",
  "url": "https://example.com",
  "sameAs": [
    "https://en.wikipedia.org/wiki/Example_Corp",
    "https://github.com/examplecorp",
    "https://www.linkedin.com/company/examplecorp"
  ]
}
</script>

The failure mode to avoid is markup that contradicts the visible page — a dateModified of last week on a page untouched since 2023, or offers listing a price the pricing page does not show. Contradiction is worse than absence, because it gives a synthesiser two sources of truth for the same claim. There is more on getting the markup itself right in structured data and rich results.

You will also see advice to publish an llms.txt file. It is cheap and harmless, but be clear-eyed about its status — llms.txt explained covers who has and has not committed to reading it, and the short version is "almost nobody, yet."

What this means for the work

The ordering matters, because the first two items gate everything else:

  1. Confirm fetchability for each agent you care about, at the edge and in robots.txt. Confirm the response body contains prose without JavaScript.
  2. Confirm the page returns fast and directly — no redirect chain, no interstitial, no 3-second TTFB.
  3. Rewrite the opening of every section so the answer precedes the discussion.
  4. De-pronoun your paragraphs so each stands alone.
  5. Add entity markup, accepting that the evidence for its effect on citation is indirect.

Most of that list is ordinary technical SEO hygiene, which is the genuinely useful conclusion here: there is no separate "AI SEO" discipline that supersedes crawlability, clean HTML, fast responses and honest structure. There is the same discipline, with the passage rather than the page as the unit of value.

If you want the fetchability and structure checks run for you rather than by hand, SEO Fix Agent audits a page's technical, meta, content and GEO signals — including robots.txt directives and whether server-rendered content is actually present — and its Prompt Studio compiles the findings into a severity-ordered prompt you can paste straight into Claude Code or Cursor to make the fixes. It audits one page per run, so point it at the pages you most want quoted.

Finally, keep expectations calibrated. Citation is noisy: the same question asked twice can produce different sources, and engines change their retrieval stacks without announcement. Track whether you are being cited, treat any single result as a sample of one, and do not rebuild your content strategy around a screenshot.

#ai-search#generative-engine-optimization#robots-txt#crawlers#content-structure

Audit your page, then ship the fix

SEO Fix Agent runs 30+ technical, content and AI-search checks on a page, then compiles every finding into a severity-ordered prompt your coding agent can execute. 75 free credits, no card.

Start free →

Keep reading