OAI-SearchBot, Robots.txt, and Why Most Brands Are Invisible to ChatGPT

OpenAI Ads · Visibility audit

Open your site's robots.txt file. Search for the string `OAI-SearchBot`. If it is not there, your products are visible to ChatGPT by default, which is good. If it is there as a `Disallow`, your products are structurally invisible to every ChatGPT recommendation, including the paid ones you are about to bid on. We have seen brands spend six figures on OpenAI Ads while the bot that delivers the recommendation is blocked at the file that takes ten seconds to fix.

This is the most common visibility failure on the channel, and it does not show up in any analytics tool. Your CPC numbers look normal, your spend draws down, your CAC reads high, and the cause is a one-line config decision made years ago to block AI crawlers by default. This post is the five-minute audit every DTC brand should run before spending another dollar.

OAI-SearchBot is not GPTBot. The distinction matters.

OpenAI operates two crawlers and they do different jobs. `GPTBot` collects data to train OpenAI's foundation models. `OAI-SearchBot` indexes pages for retrieval-time use in ChatGPT, including in shopping responses and ad recommendations. Brands that read the AI-crawler-blocking discourse from 2023-2024 often defaulted to blocking both, on the reasonable theory that they did not want their content used to train competitors' models. That logic still applies to GPTBot. It does not apply to OAI-SearchBot.

Blocking `GPTBot` and allowing `OAI-SearchBot` is the configuration most DTC brands actually want. Your product pages do not get scraped into training data. They do get indexed for live ChatGPT recommendations. If your robots.txt blocks both indiscriminately, you opted out of the channel at the file system level. The OpenAI merchant documentation covers the distinction publicly, but most operators have not read it because the merchant docs only landed in late 2025.

The three-line robots.txt audit

Run this against your live site's robots.txt right now. Replace `yourbrand.com` with your domain.

curl -s https://yourbrand.com/robots.txt | grep -i 'oai-searchbot\|gptbot\|user-agent'

Read the output carefully. The right configuration looks like this:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Three things to check. First, OAI-SearchBot is explicitly allowed. Second, GPTBot is explicitly disallowed if you do not want your content used in foundation-model training. Third, there is no blanket `User-agent: ` with a `Disallow: /` above either block that would override them. The wildcard rule applies to any bot not matched by a more specific rule; if your file has `User-agent: ` followed by `Disallow: /`, every crawler including OAI-SearchBot reads that as a hard no.

If you are using Cloudflare or another CDN-level bot-management product, this is the second place visibility leaks. Many bot-management defaults block `OAI-SearchBot` as part of the broader AI-crawler category. Check your Cloudflare bot-management rules and explicitly allow OAI-SearchBot at the WAF level. The robots.txt fix is necessary but not sufficient if the CDN is dropping the request before it reaches your origin.

JSON-LD Product schema and server-side rendering

If robots.txt is the binary switch, JSON-LD Product schema with server-side rendering is the quality dial. The crawler can reach your page. What it sees when it gets there is the next problem.

OAI-SearchBot reads structured product data the same way every other modern crawler does, through the JSON-LD blob embedded in the page HTML. If your product pages are client-rendered (React or Vue mounted into an empty div), the crawler sees the empty shell and walks away with nothing usable. Server-side rendering is not optional for products you want surfaced. Next.js App Router does this by default. Older SPA implementations often do not.

The minimum JSON-LD payload OpenAI's indexer benefits from includes the product name, brand, description, image, price, availability, GTIN where applicable, and review aggregate. The OpenAI merchant page is explicit about the fields they parse. Missing fields do not block indexing, but they meaningfully reduce the likelihood of a recommendation against a competitor whose schema is complete. Schema completeness is one of the variables that turns up in the math piece's decision rules under the Amazon-presence section, because brands with poor schema lose to Amazon listings whose schema is always complete.

How to verify your visibility (the practical check)

After the robots.txt fix and the JSON-LD audit, the only honest test of visibility is to ask ChatGPT directly. Pick five questions a buyer in your category would ask. Run them through ChatGPT. Note which brands surface, in what order, with what reasoning. Repeat the same five queries a week later. The variance gives you a sense of how stable your recommendation slot is.

Things to watch for. If competitor brands surface and yours does not despite a clean robots.txt and complete JSON-LD, the issue is likely brand authority, not technical. The model is recommending the brands it has more signal density for. Reviews, expert citations, editorial coverage, and third-party validation are the inputs that fix this; the technical baseline is necessary but not sufficient.

If your brand surfaces with stale information (old pricing, outdated SKUs, last year's positioning), your product feed is fresh but the crawler is reading a cached or out-of-date copy. Check your sitemap.xml lastmod dates and confirm your CDN cache headers are not telling crawlers to ignore updates. The technical surface for this is identical to the SEO crawl-freshness work most operators already do; the bot you are optimizing for is just different.

If you want the same loop running continuously rather than as a quarterly audit, the OpenAI Ads page shows what Cresva agents do here. The mechanical version of the check is the same; the operator just does not have to remember to run it.

Beyond robots.txt: the other invisibility traps

Four failure modes show up after the robots.txt fix is in place. Each is a meaningful drag on recommendation rate and each is fixable inside a week.

If you are walking through this list against your own site and finding two or three issues, that is normal. Most DTC brands are starting from a default-blocked or partially-blocked state because the AI-crawler discourse of 2023 told operators to block aggressively, and nobody went back to revisit the decision when OpenAI started recommending products. The five-minute audit is what catches that legacy decision before it costs you another quarter of spend.

Robots.txt is binary. JSON-LD plus SSR is the quality dial. WAF-level bot rules are the second binary switch nobody checks. Run the three audits in this post and you have done the visibility baseline that most brands paying for OpenAI Ads have not done. If you want to fold this into the same continuous monitoring loop your team already runs on Meta and Google, you can try Cresva for free.