Six DTC Stores an AI Assistant Cannot Read

While building the AI Shopper Index we tried to read 72 DTC storefronts the way an automated client reads them. Six of them would not open. Not slow, not partial, not missing a feed: 403, every request, every time.

The reflex is to call this bot blocking and move on. That reading does not survive the evidence. We sent two user agents at each store, one that announces itself as a crawler and one that claims to be desktop Chrome, and in all six cases both got the identical response.

What we sent, and what came back

Two requests per store, observed 2 August 2026. The first identifies itself honestly as our crawler. The second is an ordinary desktop Chrome string, the kind a real browser sends.

A.  Mozilla/5.0 (compatible; CresvaBot/1.0; +https://cresva.ai/bot)
B.  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36
    (KHTML, like Gecko) Chrome/126.0 Safari/537.36

Two of the six answer from Vercel and four from Cloudflare, so this is not one vendor's default. And only one of the six serves a challenge at all. That distinction is the one most write-ups of this get wrong, including our first one, so it is worth being precise about.

A challenge and a refusal are different things

BARK returns a Cloudflare managed challenge. The response carries `cf-mitigated: challenge` and the body is the familiar interstitial titled `Just a moment...`. A challenge is a door with a test on it. A real browser runs the JavaScript, passes, and continues. An agent does not, so it stops there.

The other five are flat refusals. Grovemade and Ridge answer with a block page titled `Attention Required! | Cloudflare`. Caraway and Made In answer from Vercel with no challenge page at all. Hims answers from Cloudflare, also with nothing to pass. There is no test to take. Nothing that is not already permitted gets in, browser engine or not.

Calling every 403 a bot challenge describes a default posture as hostility, and it tells the five brands with no challenge to go looking for a setting that is not there.

The part that is genuinely invisible

Three of these stores answer with a page carrying `noindex, nofollow`: BARK's challenge interstitial, and the Cloudflare block pages at Grovemade and Ridge. So a crawler is told not to index, by a page the merchant has never seen and did not write.

And because the request never reaches the origin, none of it appears in the store's own analytics. No session, no bounce, no error, no line on any dashboard. From inside the business, a store that no automated client can open looks exactly like a store nobody tried to open.

That is the same shape as the finding in our first index post: the absence produces no signal. A brand that scores zero and a brand that cannot be read both learn nothing from their own instrumentation.

How to check your own store in two commands

Send both user agents and compare. If they differ, you have user-agent filtering and it is a policy you can adjust. If they match and both are 403, the decision is being made before the user agent is read, which usually means an edge rule rather than anything in your storefront.

# A: an honest crawler string
curl -sI -A 'Mozilla/5.0 (compatible; ExampleBot/1.0)' https://YOURSTORE.com/ | head -1

# B: desktop Chrome
curl -sI -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36' https://YOURSTORE.com/ | head -1

# If both say 403, look at what answered and whether it offered a challenge:
curl -sI https://YOURSTORE.com/ | grep -iE '^(server|cf-mitigated):'

Worth running against `/products.json` and your sitemap as well as the homepage, since edge rules are often path scoped and the homepage is the least likely thing to be protected. If both requests come back clean, the next layer down is robots.txt and your CDN's bot categories, which is a separate audit with its own failure modes: the visibility baseline guide walks that one, including the four OpenAI bots and the Cloudflare category model.

What we are not saying

• Nobody here chose to block AI assistants. We have no evidence of intent and are not implying any. A 403 from an edge default is a configuration, not a decision.
• Blocking automated clients is not automatically wrong. Scraping, card testing and price harvesting are real, and these are the same controls that stop them. The trade is what it is; the point is knowing you have made it.
• This is one observation on one day. Edge configurations change, and ours is a single vantage point from a single network. Both commands above let you check us.
• These six are not scored in the index. They carry no visibility number because there is nothing to measure, which is a different thing from scoring zero. They are listed rather than dropped, because dropping them would hide the finding.

The uncomfortable version of this is short. Six recognisable DTC brands, with real budgets and real traffic, cannot be read by the software that increasingly answers the question their customers ask first. Nothing in their own reporting will tell them, and the page turning agents away is one they have never seen.