OpenAI Crawlers: Four Bots and What Each Does

Chapter 1How Many Crawlers Does OpenAI Run?

Four, and they do different jobs. As of 3 August 2026, OpenAI documents OAI-SearchBot, OAI-AdsBot, GPTBot and ChatGPT-User. Treating them as one category, which is what a blanket AI-bot rule does, is the single most expensive mistake available here: it takes a defensible decision about training data and silently applies it to the bot that decides whether ChatGPT can recommend you at all.

Roster and tokens per OpenAI's crawler documentation, developers.openai.com/api/docs/bots, retrieved 3 August 2026. This roster moves: it went from two documented bots to four inside a year, so check the source rather than any article, including this one, before acting on it.

Chapter 2Which Ones Obey Robots.txt?

Not all of them, and the exception is the reason a single blanket rule cannot express what you want. OpenAI's table marks OAI-SearchBot and GPTBot as honouring robots.txt. For ChatGPT-User it says plainly that because these actions are initiated by a user, robots.txt rules may not apply. That is a different contract, and it is the one people are surprised by.

The practical consequence is that robots.txt governs your exposure to crawling, not your exposure to ChatGPT. A user who asks ChatGPT about your product can cause a fetch that robots.txt does not control. If your reason for blocking is legal or contractual rather than about crawl budget, robots.txt is the wrong instrument and you want a server-side or WAF rule.

Chapter 3How Do You Verify a Bot Is Really OpenAI?

By IP, not by user agent. A user-agent string is a claim anyone can make, and impersonating a well-known crawler to get past bot rules is routine. OpenAI publishes a live IP range per bot, which makes verification a lookup rather than a guess.

Endpoints as documented on OpenAI's crawler page, retrieved 3 August 2026. Two things follow from these existing. First, traffic claiming to be OAI-SearchBot from outside the published ranges is not OpenAI, and treating it as OpenAI is how a scraper gets whitelisted. Second, if you are going to allow these bots past a WAF, allow them by verified IP rather than by user-agent string, or the allowance is available to anyone who types the token.

Chapter 4What Should Your Robots.txt Actually Say?

The configuration most DTC brands actually want separates training from retrieval. Block the training crawler if that is your policy, and explicitly allow the two that decide whether you can be found and advertised. Distinct User-agent groups, distinct rules, and the explicit Allow blocks placed above any wildcard rule so a broad Disallow does not capture them.

1. 1

   Search your robots.txt for OAI-SearchBot

   If the token does not appear at all, you are visible to ChatGPT by default, which is the right state. If it appears under a Disallow, you are structurally invisible to every ChatGPT recommendation including paid ones, and no amount of spend changes that.

2. 2

   Give each bot its own User-agent group

   A single group listing several agents, or a blanket AI-bot rule, cannot express the split you want. Blocking training while allowing retrieval requires two groups, not one.

3. 3

   Add an explicit Allow for OAI-AdsBot if you run ads

   Given the documentation does not state this bot's robots.txt position, an explicit Allow costs nothing and removes one variable if an ad review stalls. It is not a guarantee, which is why the next step exists.

4. 4

   Confirm against your logs, not your intent

   Grep your access logs for each of the four tokens over a week. The presence or absence of real requests is the only evidence your file is doing what you meant, and it is the step that catches a CDN silently overriding you.

The same split exists for other assistants and is worth setting at the same time. Anthropic documents three bots, ClaudeBot for training, Claude-SearchBot for search indexing and Claude-User for user-initiated fetches. Perplexity documents two, PerplexityBot for indexing and Perplexity-User, which Perplexity states generally ignores robots.txt. Both rosters were read from those vendors' own documentation on 3 August 2026 and both are worth re-checking on the same cadence as OpenAI's.

Chapter 5Why Your CDN Can Override All of It

Because a CDN refuses the request before your origin is involved, so robots.txt is never read. This is the failure that survives a correct robots.txt, and it is invisible in your own analytics for the same reason: the request never reaches you, so it produces no session, no error and no line on any dashboard.

The trap is that this is a change to defaults, not a retroactive fix. A domain that enabled the older blanket rule any time after summer 2024 is still running whatever that rule was set to, and that rule predates the split between the search and training crawlers entirely. Open the dashboard and read the Search category directly rather than assuming the July change reset it for you.