How the brand index is measured

Every question is derived from the store's own public signals: the merchant's product-type taxonomy first, then tags, then metafield values, then product titles, each weighted by how much it reveals about what is actually sold. Harvested terms are matched against curated lexicons of product nouns, attributes and use cases, ranked by accumulated weight, and used to fill question templates. No model is involved. The same code produces the question set for every brand.

The questions are never branded. A question that names the brand guarantees a mention and makes the score meaningless. What is being measured is whether an assistant surfaces you to somebody shopping for what you sell, so the question has to be about what you sell and nothing else.

Why the fixed templates were abandoned

The previous method had nine category buckets. Every store classified as fashion was asked about sustainable clothing, DTC denim, basic t-shirts and affordable jewelry. Allbirds sells shoes, socks, apparel and underwear, so three of those four questions were about categories it does not stock, and not one was about footwear. It scored 0 of 8, and the report said AI assistants do not recognise or recommend the brand. That claim was disproved in under a minute by asking about sneakers.

The score was not miscalibrated. The questions were wrong, and a wrong question produces a confident zero that looks exactly like a real one. Stores whose signals are too thin to rank a product noun still fall back to generic templates, and the run is marked as having used them so the report can say the questions were generic rather than pretending otherwise.

"We asked enough assistants and you were not named" and "we could not ask" are different findings. Only the first is a fact about the brand. The second is a fact about our run, and reporting it as a low score would be a fabrication with a number attached.

So a visibility score is only produced when at least 2 assistants answered at least 4 questions. Below that bar the run reports no score at all. A zero on the index therefore always means the questions were asked, on topic, enough times, and the brand did not come back. A brand we could not measure is omitted from the ranking and counted underneath it, never ranked last.

The same rule governs everything else the engine reports. A check that could not run is not-assessed, which is removed from the denominator rather than scored zero, so a store is never marked down for our network, our timeout, or our expired credential.

A zero means the questions were asked and the brand did not come back. It does not mean the brand never appears. Every measurement has a floor below which a real rate is indistinguishable from absence, and we publish that floor per brand rather than leaving a reader to assume it is zero.

The same question goes to every assistant, so their answers are not independent observations. We measured how much they cluster across 4,329 answered questions and 1,196 question groups: the intra-cluster correlation is 0.594, which discounts a run’s observations by a factor of 2.556. A brand asked four questions of four assistants therefore has about six effective observations, and the smallest rate that could be told apart from zero at 95% confidence is 38%. Each brand page states its own figure, because the number depends on how many questions that catalogue supported.

This is the least flattering number on the site and it is the one that makes the rest of it usable. A ranking that does not say what it could not have seen is asking to be read as more precise than it is.

Tracking a brand over time and measuring it well are different jobs, and they want opposite things. A trend line needs a constant: if the questions change between runs, a movement in the number could be the brand or could be us. A measurement wants to stay current: catalogues change, and a question set frozen forever slowly stops describing what a store sells.

So we do both. Four questions per brand are fixed at its first run and never change; the change column is computed on those alone, so it is stable by construction. Every other question sits in a pool that grows over time. Adding to the pool retires nothing and moves no history, because the core it would have to disturb is untouched and a run stays comparable to an earlier one on the questions both asked.

Roughly a third of measured brands have catalogues too thin to support the full four, and those brands say so on their own page rather than presenting a two-question trend as though it were a four-question one.

These are not edge cases we intend to fix. They are accepted limits of a question set built from a store’s own published taxonomy, and we would rather a reader know them than discover them.

• A small line can outrank the main one. We read what a catalogue declares, not how much of it sells. A skincare brand with a few fragrances can be asked about colognes. Every check we have passes, because cologne really is in the catalogue; what we cannot see is that it is a rounding error of the business.
• Price anchors follow the catalogue, including its bundles. A question can read “under $725” when a product line is full of multi-item kits. The figure is the true median of that line and still reads as absurd.
• An attribute can be real and still be odd. We require an attribute to appear on a product of the type being asked about, which removed questions like “the best fleece bags” from a laptop-bag company. It does not stop a true but unrepresentative pairing, such as asking a key-organiser brand about insulated bags.
• Eleven brands cannot be measured at all. Their published taxonomy names a flavour, a container or a merchandise line rather than a product, or the store is too small to read. We report them as unmeasured rather than scoring them, and for most of them the fact itself is the finding: a catalogue that does not name what it sells is hard for an assistant to enumerate too.

Every measurement carries the conditions it was taken under: which assistants answered, which did not, how many questions were scored. Not as a disclaimer, as part of the result. Two scores taken under different coverage are not the same kind of number, and a reader who cannot see the difference has no way to know that.

This matters most when it is inconvenient. A vendor outage on our side narrows coverage, and the honest response is to say so on the affected runs rather than publish a thinner measurement that looks identical to a full one. The index refuses to refresh at all when fewer assistants are reachable than the bar requires, because a ranking that reshuffles with our vendor quota is measuring our vendor quota.

The engine reports what tracking is visible in a page's source. That is a useful thing to know and a disqualifying thing to rank on, because it measures visibility of tracking rather than quality of tracking, and those two run in opposite directions as a team matures.

Moving tags into a server-side container is a step up in sophistication: better data quality, less client weight, more control over what is shared. It also removes those tags from the page source. A brand that does the more advanced thing scores lower. On the first brands measured, the lowest tracking score in the set belonged to a store whose tracking is almost certainly better than the highest.

So the column is reported and never ranked. A metric that falls as competence rises is not a metric, and publishing a league table ordered by it would be publishing the inversion as a finding.

There used to be one. It read 41 for Allbirds and 60 for a 46-product indie store, and that was not a calibration problem to be tuned. It was measuring the wrong thing and being framed as another.

The components share no unit. Schema completeness, pixel hygiene, homepage conversion structure and share of assistant recommendation are four different questions, and a weighted mean asserts both that they are commensurable and that ten points of one trade against ten points of another. They are not and it does not.

Worse, it inverted on sophisticated brands by construction. Maturity moves work out of static HTML: tags into server-side containers, rendering into a SPA, content behind a CDN edge. Every one of those lowers what a crawler observes while raising actual capability. Coverage also varied per run, so averaging over whichever collectors happened to succeed made two runs of the same store non-comparable, never mind two different stores.

And it invited exactly the misreading the reports shipped. "Overall audit score of 41 out of 100" reads as "this business is failing", when what was measured was "this homepage exposes little to a document fetch". A reader can act on "0 of 12 schema types present". Nobody can act on 41. The index ranks one named column for the same reason.

Every class below is a real defect this engine shipped. They are published because a measurement nobody can check is a claim, and because the four instances of the first class were found separately and looked like four unrelated bugs until they were put side by side, at which point they were obviously one mistake made four times.

The last item in each entry is the point of this document. An error that has been fixed will recur; an error with an invariant behind it has to get past the invariant first.

Every index run appends a row that is never updated and never deleted, enforced by the database rather than by convention. It carries each collector score, the coverage conditions, the raw finding counts, and the complete text of every assistant answer.

The full answers are kept rather than excerpts because the matcher has already improved once and will again. An excerpt cut around today's match freezes the record at today's matching quality and makes a future correction impossible. Keeping the whole answer means a past zero can be re-examined against a better method instead of being taken on trust.

The refresh runs 3 times a day and takes the stalest brands each time. No brand is re-measured inside 6 days, so any one brand is re-measured about weekly. The floor, not the schedule, is what holds that interval. Re-running the same brand daily would render assistant run-to-run variance as brand movement, which is the same class of false signal as every failure mode above: a chart of noise is still a fabricated finding. Sampling different brands each time is not that, and it is what keeps the interval from stretching as the registry grows.

If a measurement here is wrong about your brand, the useful thing to send is the question we asked and what your own check returned. Both are on your report page, which lists every question put to every assistant. A correction that changes the method bumps the version at the top of this document, and a correction that changes what a score means bumps the rubric instead.