Skip to content
PageSoarPageSoar

How to get cited by ChatGPT search: start with one line in robots.txt

OpenAI documents which crawler feeds ChatGPT's search answers. If it is blocked, nothing else matters.

Author: HD4 min read

Table of contents
  1. Key takeaways
  2. Two crawlers, two jobs
  3. Check yours
  4. Then write pages worth quoting
  5. What PageSoar reads
  6. See it on your own site
  7. Sources

Two crawlers, two jobs#

OpenAI's bots page lists OAI-SearchBot, used to surface websites in ChatGPT's search features, and GPTBot, used for training. Each setting in robots.txt is independent, so you can allow one and block the other.

Check yours#

Open your site's robots.txt and look for a Disallow under OAI-SearchBot or under a wildcard user agent. OpenAI recommends allowing OAI-SearchBot if you want to appear in search results.

Then write pages worth quoting#

A direct answer near the top, a named source for every number, and a title that says what the page is.

What PageSoar reads#

The free website check reads robots.txt and names any AI crawler it blocks. The AI citations report then shows, search by search, whether ChatGPT names you. What PageSoar does for SEO and AI answers lists the rest.

See it on your own site#

The free website check reads a site you name and reports what holds its pages back, with no account and no card. It changes nothing on your site.

Run the free website check

Sources#