1. Questions come from the catalogue, not from a template
Every question is derived from the store's own public signals: the merchant's product-type taxonomy first, then tags, then metafield values, then product titles, each weighted by how much it reveals about what is actually sold. Harvested terms are matched against curated lexicons of product nouns, attributes and use cases, ranked by accumulated weight, and used to fill question templates. No model is involved. The same code produces the question set for every brand.
The questions are never branded. A question that names the brand guarantees a mention and makes the score meaningless. What is being measured is whether an assistant surfaces you to somebody shopping for what you sell, so the question has to be about what you sell and nothing else.
Why the fixed templates were abandoned
The previous method had nine category buckets. Every store classified as fashion was asked about sustainable clothing, DTC denim, basic t-shirts and affordable jewelry. Allbirds sells shoes, socks, apparel and underwear, so three of those four questions were about categories it does not stock, and not one was about footwear. It scored 0 of 8, and the report said AI assistants do not recognise or recommend the brand. That claim was disproved in under a minute by asking about sneakers.
The score was not miscalibrated. The questions were wrong, and a wrong question produces a confident zero that looks exactly like a real one. Stores whose signals are too thin to rank a product noun still fall back to generic templates, and the run is marked as having used them so the report can say the questions were generic rather than pretending otherwise.
2. An unconfirmed absence never scores
"We asked enough assistants and you were not named" and "we could not ask" are different findings. Only the first is a fact about the brand. The second is a fact about our run, and reporting it as a low score would be a fabrication with a number attached.
So a visibility score is only produced when at least 2 assistants answered at least 4 questions. Below that bar the run reports no score at all. A zero on the index therefore always means the questions were asked, on topic, enough times, and the brand did not come back. A brand we could not measure is omitted from the ranking and counted underneath it, never ranked last.
The same rule governs everything else the engine reports. A check that could not run is not-assessed, which is removed from the denominator rather than scored zero, so a store is never marked down for our network, our timeout, or our expired credential.
3. Coverage is stated on every run
Every measurement carries the conditions it was taken under: which assistants answered, which did not, how many questions were scored. Not as a disclaimer, as part of the result. Two scores taken under different coverage are not the same kind of number, and a reader who cannot see the difference has no way to know that.
This matters most when it is inconvenient. A vendor outage on our side narrows coverage, and the honest response is to say so on the affected runs rather than publish a thinner measurement that looks identical to a full one. The index refuses to refresh at all when fewer assistants are reachable than the bar requires, because a ranking that reshuffles with our vendor quota is measuring our vendor quota.
4. Tracking observability cannot rank brands
The engine reports what tracking is visible in a page's source. That is a useful thing to know and a disqualifying thing to rank on, because it measures visibility of tracking rather than quality of tracking, and those two run in opposite directions as a team matures.
Moving tags into a server-side container is a step up in sophistication: better data quality, less client weight, more control over what is shared. It also removes those tags from the page source. A brand that does the more advanced thing scores lower. On the first brands measured, the lowest tracking score in the set belonged to a store whose tracking is almost certainly better than the highest.
So the column is reported and never ranked. A metric that falls as competence rises is not a metric, and publishing a league table ordered by it would be publishing the inversion as a finding.
5. Why there is no overall score
There used to be one. It read 41 for Allbirds and 60 for a 46-product indie store, and that was not a calibration problem to be tuned. It was measuring the wrong thing and being framed as another.
The components share no unit. Schema completeness, pixel hygiene, homepage conversion structure and share of assistant recommendation are four different questions, and a weighted mean asserts both that they are commensurable and that ten points of one trade against ten points of another. They are not and it does not.
Worse, it inverted on sophisticated brands by construction. Maturity moves work out of static HTML: tags into server-side containers, rendering into a SPA, content behind a CDN edge. Every one of those lowers what a crawler observes while raising actual capability. Coverage also varied per run, so averaging over whichever collectors happened to succeed made two runs of the same store non-comparable, never mind two different stores.
And it invited exactly the misreading the reports shipped. "Overall audit score of 41 out of 100" reads as "this business is failing", when what was measured was "this homepage exposes little to a document fetch". A reader can act on "0 of 12 schema types present". Nobody can act on 41. The index ranks one named column for the same reason.
6. Failure modes
Every class below is a real defect this engine shipped. They are published because a measurement nobody can check is a claim, and because the four instances of the first class were found separately and looked like four unrelated bugs until they were put side by side, at which point they were obviously one mistake made four times.
The last item in each entry is the point of this document. An error that has been fixed will recur; an error with an invariant behind it has to get past the invariant first.
a. Engine failure attributed to the merchant
Something on our side did not work, and the report described that as a fact about the store.
What was published
- Allbirds was told it had no Product schema, so AI agents could not parse its product data. We had scanned the homepage, and homepages do not carry Product schema. The schema was on the product pages the whole time.
- A store was told it refuses automated browsers. What actually happened was that our screenshot vendor got rate-limited by the store's CDN, on the vendor's shared IP pool. The same store answered our own crawler with a 200 on every attempt.
- Reports said the Meta Ad Library returned no usable response, which reads as a fact about the merchant's ad account. Our own access token had been dead for 182 days.
- A store was told ClaudeBot was blocked when it was not. The check fell through from ClaudeBot to the deprecated anthropic-ai token, so blocking a string ClaudeBot does not honour was reported as blocking ClaudeBot.
Root cause
A failed or mis-aimed observation was allowed to produce a finding. None of these sentences said 'you' and none of them said 'us' either, so a reader supplied the subject, and the subject a reader supplies is always themselves.
Invariant that now prevents it
A collector may emit a claim about a store only from an observation that succeeded, on the page that observation belongs on. When an observation fails or was never attempted, it emits a coverage statement that affirmatively locates the failure with us. Not-assessed is a third state, distinct from pass and from fail, and it is removed from the denominator rather than scored zero.
Enforced by lib/growth/merchant-claim.ts, validated over every failure-path reason each collector can emit by the compliance suite in __tests__/growth-engine.test.ts.
b. Unverified success credited to the merchant
The opposite direction, and the one that is easier to miss because nobody complains about a score that is too high.
What was published
- og:image scored on the presence of the tag. The tag is on virtually every modern site, so the points were close to free, and a page whose card renders blank in Slack scored the same as one that renders.
- canonical scored the same way: the tag exists, so the URL inside it was assumed to resolve.
- llms.txt was worth more than any other single item and was awarded on an HTTP 200. A catch-all rewrite answers 200 for a file that does not exist.
- sitemap presence was inferred from a 200 rather than from parsing what came back.
Root cause
Presence of a pointer was treated as proof of the thing pointed at. Every one of these is a claim we never actually checked, and each inflates the score of a site that would fail in the real world.
Invariant that now prevents it
A check may return ok only when it observed the thing it claims, and not-ok only when it observed the absence. Following a pointer is part of the check, not an optimisation. Anything else is not-assessed. Concretely, two questions have to be answered no when a check is added: if our network is broken, does this blame the merchant, and if their asset is broken, does this still award points.
Enforced by The three-state LinkedAssetCheck shape in lib/growth/collectors/ai-readiness.ts, which replaced the booleans, plus the linked-asset verification pass that actually fetches og:image and canonical targets.
c. Aggregation error
The arithmetic itself was wrong, in a way that was visible on the page.
What was published
- A report stated '104 of 102 product titles' had an issue. A count larger than the population it is drawn from.
- The same store was told '61 of 102 products have an image_link problem: Missing product image'. The ground truth from its own products.json was six missing images and about fifty-five reused ones.
Root cause
Issues were grouped by field rather than by field and issue type, and counted as issues rather than as distinct products. A product with two title problems counted twice, which is where a number larger than the population came from. Collapsing four different image checks into one bucket kept the total and described it with whichever issue happened to be seen first.
Invariant that now prevents it
Any count presented as 'N of M' counts distinct members of M, and the label describes the specific issue type being counted rather than the field it belongs to. A total that can exceed its own population is a bug by construction, not a rounding question.
Enforced by Grouping by field and issue type with a distinct-product Set in lib/growth/collectors/catalogue-quality.ts.
d. Prose leak
The invariant was enforced where it was written and not where the text was actually generated.
What was published
- Collector reasons were validated against the merchant-claim rule. The synthesised report was not, so a model writing prose from those collectors could reintroduce exactly the claim the validator existed to catch, in its own words.
- Vendor detail reached public copy through the same gap: HTTP statuses, JSON error bodies, request ids and billing text are all things a collector sanitises at source and a generated paragraph can reproduce.
Root cause
The validated surface and the published surface were different surfaces. Structured fields were checked; the paragraph a visitor actually reads was not.
Invariant that now prevents it
Generated prose is validated against the same invariant as the structured reasons it is written from, using the set of collectors whose measurement did not complete, including any collector that returned ok without clearing its coverage bar. Vendor detail is replaced wholesale at a single chokepoint rather than patched, because a partial scrub is a leak waiting to be re-found.
Enforced by validateSynthesisProse in lib/growth/merchant-claim.ts, and the single publicReason chokepoint in lib/growth/public-reason.ts shared by the report builder and the PDF builder.
e. Matching error
The measurement was taken correctly and then compared wrongly, which produces a confident zero.
What was published
- Ten Thousand scored 0 on a run where assistants had in fact recommended it. The brand name is derived from a domain label and domains have no spaces, so we were looking for 'Tenthousand' while every assistant wrote 'Ten Thousand'.
- Catalogue text arrives HTML-encoded. A literal apostrophe and ' are different strings, so terms harvested from product data failed to match the same words written normally.
Root cause
One canonical form was assumed on both sides of a comparison whose two sides come from different systems: a domain label against natural prose, encoded markup against decoded text.
Invariant that now prevents it
Any comparison between our derived form and third-party text normalises both sides first, and a name match permits the separators a human would insert while still requiring every character in order and a non-alphanumeric boundary at each end. A zero is only reported when the matcher has been shown to be capable of a hit, and the answers that produced it are stored so the comparison can be re-run when the matcher improves.
Enforced by brandPatterns in lib/growth/collectors/ai-visibility.ts, entity decoding in lib/growth/query-generator.ts, and the full answer text retained in BrandIndexSnapshot.probeDetail so history can be re-scored rather than trusted.
7. The record
Every index run appends a row that is never updated and never deleted, enforced by the database rather than by convention. It carries each collector score, the coverage conditions, the raw finding counts, and the complete text of every assistant answer.
The full answers are kept rather than excerpts because the matcher has already improved once and will again. An excerpt cut around today's match freezes the record at today's matching quality and makes a future correction impossible. Keeping the whole answer means a past zero can be re-examined against a better method instead of being taken on trust.
Brands are refreshed weekly. Daily would render assistant run-to-run variance as brand movement, which is the same class of false signal as every failure mode above: a chart of noise is still a fabricated finding.
8. Corrections
If a measurement here is wrong about your brand, the useful thing to send is the question we asked and what your own check returned. Both are on your report page, which lists every question put to every assistant. A correction that changes the method bumps the version at the top of this document, and a correction that changes what a score means bumps the rubric instead.