OAI-SearchBot and Robots.txt: Visibility Baseline

Chapter 1The Five-Minute Audit

Open your site's robots.txt file. Search for the string OAI-SearchBot. If it is not present, your products are visible to ChatGPT by default. That is the right state. If OAI-SearchBot is present under a Disallow directive, your products are structurally invisible to every ChatGPT recommendation, including the paid ones you are about to bid on. Brands have spent six figures on OpenAI Ads while the bot that delivers the recommendation was blocked at the file that takes ten seconds to fix.

This is the most common visibility failure on the channel, and it does not show up in any analytics tool. CPC numbers look normal. Spend draws down. CAC reads high. The cause is a one-line config decision made years ago to block AI crawlers by default. The validator below evaluates your live file in place. Paste, read the per-crawler verdict, fix the one that matters.

Chapter 2OAI-SearchBot Is Not GPTBot

OpenAI documents four bots and they do different jobs. Treating them as one category is the mistake that costs brands the channel. Blocking the training crawler, GPTBot, is a defensible decision. Blocking the search crawler alongside it, OAI-SearchBot, which is what most blanket rules do, removes you from ChatGPT recommendations entirely.

Two of those rows deserve more than a table cell. The first is OAI-AdsBot. If you are running or about to run OpenAI Ads, this is not a footnote. OpenAI uses it to validate the safety of pages submitted as ads, so a bot rule or WAF policy that blocks it can stall your ads at review while your robots.txt looks correct for search. Brands audit for OAI-SearchBot and stop, because until recently that was the whole story.

The second is ChatGPT-User, and it is the reason robots.txt is not a single switch across all four. OpenAI is explicit that because those actions are initiated by a user, robots.txt rules may not apply to it. So robots.txt governs the three crawling bots; the user-initiated fetch is a different contract. Bot table and robots.txt behaviour per OpenAI's crawler documentation, developers.openai.com/api/docs/bots, retrieved 3 August 2026.

Brands that read the AI-crawler-blocking discourse from 2023 to 2024 often defaulted to blocking everything, on the reasonable theory that they did not want their content used to train competitors' models. That logic still applies to GPTBot. It does not apply to OAI-SearchBot, and it actively works against you on OAI-AdsBot if you are buying the channel.

Chapter 3The CDN-Level Trap

Robots.txt is necessary but not always sufficient. The second place visibility leaks is the CDN. Many bot-management defaults at Cloudflare, Akamai, and Fastly classify OAI-SearchBot as part of the broader AI-crawler category and block it at the WAF layer regardless of what your robots.txt says.

The check is straightforward. Find your CDN's bot-management or WAF rules. Look for any rule that blocks OAI-SearchBot, GPTBot collectively, or “AI bots” as a category. Explicitly allow OAI-SearchBot. The robots.txt fix means nothing if the CDN drops the request before it reaches your origin.

This is not hypothetical. Six recognisable DTC brands in our public registry answer every request with a 403, from a crawler user agent and a desktop Chrome user agent alike, with the recorded evidence brand by brand: six DTC stores an AI assistant cannot read.

Chapter 4JSON-LD and Server-Side Rendering

Robots.txt is the binary switch. JSON-LD Product schema markup with server-side rendering is the quality dial. The crawler can reach your page. What it sees when it gets there is the next problem.

OAI-SearchBot reads structured product data the same way every modern crawler does, through the JSON-LD blob embedded in the page HTML. If your product pages are client-rendered (React or Vue mounted into an empty div), the crawler sees the empty shell and walks away with nothing usable. Server-side rendering is not optional for products you want surfaced. Next.js App Router does this by default. Older SPA implementations often do not.

Missing fields do not block indexing, but they reduce the likelihood of a recommendation against a competitor whose schema is complete. The product-feed optimization guide covers the schema-completeness work in detail.

Chapter 5Verifying Visibility

After the robots.txt fix and the JSON-LD audit, the only honest test of visibility is to ask ChatGPT directly. Pick five questions a buyer in your category would ask. Run them through ChatGPT. Note which brands surface, in what order, with what reasoning. Repeat the same five queries a week later. The variance gives you a sense of how stable your recommendation slot is.

If competitor brands surface and yours does not despite a clean robots.txt and complete JSON-LD, the issue is brand authority, not technical. The model recommends the brands it has more signal density for. Reviews, expert citations, editorial coverage, and third-party validation are the inputs that fix this. The technical baseline is necessary but not sufficient. The agent-visibility playbook covers the brand-authority side in detail.

Chapter 6Other Invisibility Traps

Four failure modes show up after robots.txt is in place. Each is meaningful drag on recommendation rate. Each is fixable inside a week.

If you are walking through this list against your own site and finding two or three issues, that is normal. Most DTC brands start from a default-blocked or partially-blocked state because the 2023 AI-crawler discourse told operators to block aggressively, and nobody revisited the decision when OpenAI started recommending products in late 2025. The five-minute audit catches the legacy decision before it costs you another quarter of spend. If your unit economics are the question, the ChatGPT Ads unit-economics guide runs the math.