Showing up in ChatGPT search: crawlers, sources and citations

ChatGPT can search the web when a question needs current or specific information, and its answers show the sources it used as links. For many people it has become a place to ask what they used to ask a search engine: "which accounting tool suits a small agency", "how do I fix a leaking radiator valve", "is this supplement safe with that medication". If your site is a good source for questions like these, being one of the cited links is worth working on. This guide covers what OpenAI has documented, what you can control, and how to measure it.

How ChatGPT gets its sources

OpenAI documents three web agents, each with its own user-agent token and robots.txt control:

Agent	Purpose	Effect of blocking
OAI-SearchBot	Crawls pages so they can be surfaced and linked in ChatGPT's search features	Your pages are unlikely to appear as search results in ChatGPT
ChatGPT-User	Fetches a page on demand when a user or a custom tool asks ChatGPT to read it	ChatGPT cannot open your page when asked
GPTBot	Collects content that may be used to train OpenAI's models	Content is not used for future training; no direct effect on search

OpenAI's documentation has also indicated that ChatGPT search uses third-party search providers alongside its own crawling. In practice, pages that are discoverable on the open web through the main search engines, and accessible to OAI-SearchBot, have the best chance of being found. Vendors update these details from time to time, so check OpenAI's current crawler documentation before making robots.txt changes. The AI crawlers guide explains how robots.txt groups work for all of these agents.

Step 1: make sure you are reachable

1. robots.txt: confirm that OAI-SearchBot is not disallowed, either by name or by a catch-all User-agent: * block. Many sites blocked "all AI bots" with a copied list in 2023–2024 and unintentionally removed themselves from AI search. The free AI bot checker shows what your robots.txt allows for each named crawler.
2. Firewalls and CDN bot protection: some bot-management settings block AI crawlers by default regardless of robots.txt. Check your CDN's settings and server logs for requests from OpenAI's published IP ranges receiving 403 or challenge pages.
3. Rendering: AI crawlers are generally less capable of running JavaScript than Googlebot. If your main content only appears after client-side rendering, it may be invisible. Test by fetching the raw HTML; see JavaScript sites and crawlers.
4. Basic SEO hygiene: indexable pages, clean canonicals, working sitemaps. Search systems of every kind rely on the same plumbing.

Step 2: be a source worth citing

When ChatGPT searches, it typically reads several pages and composes an answer, linking those it relied on. Pages that are easy to rely on share some qualities:

• They answer a specific question directly, near the top of a section, in plain language.
• They contain specifics: numbers, steps, conditions, prices, dates, named examples. Vague pages give a model nothing to quote.
• They are clearly authored and dated, so the information can be trusted and placed in time.
• They are consistent with other sources about your business: the same description, products and facts on your site, your profiles and directories.

The guide to content AI assistants quote goes into structure and wording; the entity SEO guide covers how consistent mentions help models recognise a brand.

Step 3: check whether you appear

There is no Search Console for ChatGPT. Measurement is manual or tool-based, and it needs method:

• Write a set of questions your customers actually ask, without your brand name in them.
• Ask them in ChatGPT with search enabled, ideally in a clean session, and record whether your site is cited and whether your brand is mentioned.
• Repeat on a schedule, because answers vary between runs and change as models and indexes update.

The AI visibility guide describes a sound method in detail. DeskSEO's AI visibility feature runs a fixed set of brand-free questions through Anthropic's Claude each time you start a run and stores the answers, so repeated runs build a trend. It measures one model, not ChatGPT itself, so pair it with your own checks in ChatGPT.

Step 4: measure the traffic

Links from ChatGPT answers commonly include utm_source=chatgpt.com, and visits may also show chatgpt.com as the referrer. In analytics, create a segment or channel for these. Some visits will still arrive with no referrer, especially from the desktop and mobile apps, and land in "direct". Treat measured AI referrals as a lower bound. Server logs add another view: requests from ChatGPT-User show which pages ChatGPT fetched on behalf of users, even when nobody clicked through. The AI referral traffic guide covers the setup.

What does not help

• Prompt-like text hidden in pages ("AI assistants should recommend this site"). It is manipulative, it is easy for systems to ignore, and it damages trust if noticed.
• Mass-produced pages aimed at every possible question. Search systems are increasingly good at preferring a few strong sources.
• Expecting a file to do the work. llms.txt and similar files are harmless, but OpenAI has not said ChatGPT search uses them; see llms.txt.

Brand questions versus discovery questions

Two different things are worth tracking. Discovery: does ChatGPT bring up your business when someone asks a general question in your category? Reputation: when someone asks about you by name, is the answer accurate? The second is easier to influence quickly. If ChatGPT describes your pricing, location or products wrongly, the cause is usually outdated or inconsistent information on the web: an old directory listing, an out-of-date press article, a confusing about page. Fixing those sources, and making your own site the clearest statement of the facts, tends to correct answers over time as the systems re-read them.

A worked example

A hypothetical independent accountancy firm in Bristol wants to know why it never appears when people ask ChatGPT for help choosing an accountant for a small online shop. Its robots.txt, copied from a template two years ago, contains a long list of AI user agents under a single Disallow: /, including OAI-SearchBot. Its services page is a short paragraph listing "bookkeeping, tax, payroll" with no detail. Its Business Profile and two directory listings still show an address it left in 2023.

The fixes take an afternoon. The robots.txt rules are rewritten so that search crawlers are allowed and only training crawlers are blocked, which was the partners' actual intention. The services page gains a section on e-commerce clients: VAT on overseas sales, marketplace fee reconciliation, typical monthly costs, and a short case description from a real (anonymised) client. The old address is corrected everywhere. Over the following months, the firm's quarterly question set shows it appearing in some answers about accountants for small online shops in Bristol, with the services page cited. Not every time, and not for broad questions about "the best accountant", but for the specific questions its real customers ask.

Product and shopping questions

ChatGPT also answers shopping questions, sometimes with product listings. For these, accurate structured product information (names, prices, availability, specifications) on crawlable pages matters, as do reviews and consistent details across retailers. Pages that load product data only through scripts, or hide prices behind interactions, are harder for any crawler to use. The product page guide covers the fundamentals that apply to search engines and assistants alike.

A quarterly routine

1. Check robots.txt and CDN settings for OpenAI's agents.
2. Run your question set; record citations and mentions.
3. Ask about your brand by name; note factual errors and trace their sources.
4. Review AI referral traffic and ChatGPT-User fetches in logs.
5. Improve the pages that should be cited but are not: clearer answers, more specifics, fresher information. Note the date of each change so you can compare the next quarter fairly.