Generative engine optimization (GEO) is the practice of making web pages easy for AI answer engines such as ChatGPT search, Perplexity, Google AI Overviews and Microsoft Copilot to crawl, understand and cite as sources. It builds on ordinary SEO: the page must be accessible to the right crawlers, rank or be retrieved for the question, and contain clear, verifiable passages that a model can quote. Nobody can guarantee that an AI system will cite a particular page.
- AI answer engines mostly cite pages they can retrieve from a search index, so crawlability and normal search visibility come first.
- Training crawlers and search crawlers are separate: you can block one and allow the other in robots.txt.
- Google-Extended controls use of your content for Gemini, not whether you appear in Google Search or AI Overviews.
- llms.txt is a proposal, not a standard, and support for it is uneven.
- Direct answers, clear headings, tables, sourced facts and visible authorship make passages easier to quote.
The term comes partly from a 2023 research paper, "GEO: Generative Engine Optimization" by researchers from Princeton and other institutions, which tested how changes such as adding citations, quotations and statistics affected visibility in generated answers on a benchmark. It is a useful framing, but results from a research setup do not translate directly into rules for any specific commercial product, and the products change often. This guide sticks to mechanisms that are documented or observable.
How AI answer engines find and cite sources
Most AI search products work in a similar way, often called retrieval-augmented generation:
- The system interprets the user's question and may break it into several search queries.
- It retrieves candidate pages from a search index, either its own or a partner's, and sometimes fetches pages live.
- It extracts relevant passages and passes them to a language model.
- The model writes an answer and attaches links to the pages it drew on.
The practical consequence is that a page usually has to be indexed and retrievable before it can be cited, and then contain a passage that answers the question cleanly. Knowledge a model absorbed during training is a different route: it can shape answers, but it does not normally produce a link back to you.
| Product | Where sources come from | What that means for you |
|---|---|---|
| ChatGPT search | OpenAI's own search crawler plus third-party search providers | Allow OAI-SearchBot; also be visible in conventional search |
| Perplexity | Perplexity's own index and live fetches | Allow PerplexityBot |
| Google AI Overviews and AI Mode | Google Search's index, crawled by Googlebot | Be indexed and eligible for snippets in Google Search |
| Microsoft Copilot | Bing's index | Be indexed by Bingbot; Bing Webmaster Tools helps |
Google states that there are no extra technical requirements for appearing in AI Overviews or AI Mode beyond being indexed and eligible to show a snippet; its AI features documentation explains this and how snippet controls such as nosnippet apply.
AI crawler user agents
AI companies use different crawlers for different purposes. The distinction that matters most is between crawlers that collect training data and those that fetch pages for search and answers.
| Token | Operator | Purpose |
|---|---|---|
GPTBot | OpenAI | Collects content that may be used to train OpenAI models |
OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search results and citations |
ChatGPT-User | OpenAI | Fetches a page when a ChatGPT user or action asks for it; not an automatic crawler |
PerplexityBot | Perplexity | Indexes pages for Perplexity answers |
ClaudeBot | Anthropic | Collects content that may be used to train Claude models; Anthropic documents separate agents for search and user-requested fetches |
Google-Extended | A robots.txt control token, not a separate crawler (see below) |
Google-Extended deserves a careful explanation. It has no user agent string of its own; Google crawls with its usual crawlers. Adding a Google-Extended rule to robots.txt tells Google whether content it crawls may be used to train future Gemini models and to ground answers in Gemini apps and Vertex AI. It does not affect whether your pages are crawled, indexed or ranked in Google Search, and it does not remove you from AI Overviews, which are part of Search. To limit how your content is shown in Search, including AI Overviews, Google points to snippet controls such as nosnippet, max-snippet and data-nosnippet.
Crawler names and behaviour change, so check each operator's current documentation before relying on a rule. A user agent string can be faked by anyone; OpenAI, Google and others publish IP ranges or verification methods so you can confirm that a request in your logs is genuine.
robots.txt rules for AI crawlers
robots.txt is a voluntary standard (RFC 9309) that well-behaved crawlers follow. A common policy is to allow search and citation crawlers while opting out of model training:
# Allow AI search and citation crawlers
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Opt out of model training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Everyone else
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml- Decide separately whether you want training use and whether you want to be cited. They are different choices.
- Remember that a crawler which finds a group naming it follows only that group and ignores the
User-agent: *rules. - Check that a firewall, CDN bot protection or rate limiter is not blocking crawlers you allowed in robots.txt. This is a frequent, silent cause of AI invisibility.
- Keep the robots.txt file small, valid and served with a 200 status.
Our AI search (GEO) checker reads your robots.txt and reports which AI crawlers are allowed or blocked, alongside other readiness checks.
What llms.txt is, and what it is not
llms.txt is a proposal, published in 2024, for a Markdown file at /llms.txt that gives language models a short summary of a site and links to its most useful pages, optionally with clean Markdown versions. It is not an IETF or W3C standard, and it does not control access the way robots.txt does.
Adoption is uneven. Some documentation sites and developer tools publish one, and some AI coding tools read them, but the major AI search engines have not committed to using it for ranking or citation, and Google has said publicly that its Search systems do not use it. Creating one is cheap and harmless, so it can be worth doing for documentation-heavy sites, but do not expect it to change whether you are cited.
# Example Co
> Example Co makes network monitoring software for small businesses.
## Docs
- [Getting started](https://www.example.com/docs/start.md): installation and first setup
- [API reference](https://www.example.com/docs/api.md): endpoints and authenticationStructuring content so it can be quoted
Answer engines extract passages, so each important section should make sense on its own. The habits that help are the same ones that make pages easier for people to scan:
- Answer first. Put a direct two- or three-sentence answer to the page's main question near the top, using the terms people actually search for.
- Descriptive headings. Use H2 and H3 headings phrased as the questions or topics they answer, with one idea per section.
- Tables and lists. Comparisons, specifications, prices and steps are easier to extract accurately from a table or numbered list than from long paragraphs.
- Specific, sourced facts. State figures, dates, names and limits precisely and link to the primary source. Vague claims give a model nothing to quote.
- Freshness. Show a visible "last updated" date and actually update the content when facts change. Many questions favour recent information.
- Author and organisation trust. Show who wrote the page and why they are qualified, and provide clear about, contact and policy pages. These signals matter to search ranking, which retrieval largely depends on.
- Structured data. Schema.org markup such as
Article,Organization,Person,ProductorBreadcrumbListhelps machines identify what a page is and who published it. It must match the visible content, and it is not a shortcut to citations.
Avoid writing for machines at the expense of readers. Keyword-stuffed or AI-padded text tends to perform poorly in the search systems that answer engines rely on, and it gives a model no better material to quote.
Technical basics
- Serve the main content in the HTML. Not every crawler runs JavaScript. If your text only appears after client-side rendering, some AI crawlers may see an empty page. Server-side rendering, static generation or pre-rendering avoids this; check by viewing the page source rather than the rendered page.
- Be fast and reliable. Live fetches often have short time limits, and slow or failing responses can mean a page is skipped. Test server response time and page weight with our website speed test.
- Get the SEO fundamentals right. Correct status codes, canonical tags, a sitemap, indexable pages and descriptive titles all feed the indexes that AI search uses. Our SEO checker audits these.
- Do not hide content behind interactions. Text in tabs that load on click, infinite scroll or login walls may not be retrieved.
Measuring AI referrals
There is no complete report of AI citations, but you can piece together a picture:
- Referrer traffic in analytics. Look for sessions from domains such as
chatgpt.com,perplexity.ai,copilot.microsoft.comandgemini.google.com. ChatGPT commonly addsutm_source=chatgpt.comto the links it shows. - Expect gaps. Clicks from mobile apps often arrive without a referrer and are counted as direct traffic.
- Google Search Console. Clicks and impressions from AI Overviews and AI Mode are included in normal Search performance data and are not reported separately.
- Server logs. Count requests from OAI-SearchBot, PerplexityBot, ChatGPT-User and similar agents, verified by IP, to see what is being crawled and fetched.
- Manual spot checks. Ask the engines your key questions periodically and record which sources they cite. Answers vary between runs, users and locations, so treat this as a sample, not a ranking.
Frequently asked questions
Is GEO different from SEO?
It is an extension of SEO rather than a replacement. AI answer engines largely retrieve from search indexes, so technical SEO and quality content remain the foundation. GEO adds attention to AI crawler access, quotable passages and measuring AI referrals.
Does blocking GPTBot remove my site from ChatGPT search?
Not by itself. OpenAI treats GPTBot (training) and OAI-SearchBot (search) separately, so you can block GPTBot and still allow OAI-SearchBot to make your pages eligible to appear in ChatGPT search results.
Will blocking Google-Extended remove me from AI Overviews?
No. Google-Extended only controls use of your content for Gemini model training and grounding in Gemini apps and Vertex AI. AI Overviews are part of Google Search and follow Search controls such as nosnippet.
Do I need an llms.txt file?
No. It is an optional proposal with uneven support. It can help some tools understand documentation sites, but there is no evidence that it is required for AI search citations.
How can I tell whether ChatGPT or Perplexity cites my site?
Check analytics for referrals from their domains, look for utm_source=chatgpt.com, review server logs for their crawlers, and ask them your target questions directly. None of these gives a complete count.