How the AI Shopper Index is measured

This engine asks questions in two different ways, for two different jobs. They are described separately below and they are not interchangeable, so it is worth being exact about which one produces the ranking on the index.

The category pool, which is what the index ranks

Each category has ONE fixed question set, written once and unbranded, and every brand in that category is measured against exactly the same questions. The set is put to every assistant on the panel, the answers are pooled, and a brand is credited when an answer names it. Nobody is asked a question chosen for them, and no brand can be ranked above another because it was asked easier questions, because there is only one set of questions in the category.

Today that is 7 categories and 594 questions in total, put to 4 assistants. Those figures are counted from the question set itself at the moment this page renders, so they cannot drift from what actually ran.

Share of voice is therefore a share OF SOMETHING: of the answers given to one category’s shared questions, by the whole panel, in that run. That is the number the index orders on.

Per-brand questions, which are the customer method

A connected brand’s own report derives its questions from its own catalogue, which is what section 1 describes and argues for. That method produces a report about one store and never produces a ranking: two brands asked different questions cannot be placed above or below each other, and doing it anyway is the defect that made the pool necessary.

So when you read section 1’s case against fixed templates, read it as an argument about the per-brand report. The index uses a fixed set on purpose, for the opposite reason: comparability between brands is the whole point of a ranking and is worth the loss of per-store fit. Section 2c, on a fixed core and a growing pool, is likewise about per-brand tracking only.

Every question is derived from the store's own public signals: the merchant's product-type taxonomy first, then tags, then metafield values, then product titles, each weighted by how much it reveals about what is actually sold. Harvested terms are matched against curated lexicons of product nouns, attributes and use cases, ranked by accumulated weight, and used to fill question templates. No model is involved. The same code produces the question set for every brand.

The questions are never branded. A question that names the brand guarantees a mention and makes the score meaningless. What is being measured is whether an assistant surfaces you to somebody shopping for what you sell, so the question has to be about what you sell and nothing else.

Why the fixed templates were abandoned

The previous method had nine category buckets. Every store classified as fashion was asked about sustainable clothing, DTC denim, basic t-shirts and affordable jewelry. Allbirds sells shoes, socks, apparel and underwear, so three of those four questions were about categories it does not stock, and not one was about footwear. It scored 0 of 8, and the report said AI assistants do not recognise or recommend the brand. That claim was disproved in under a minute by asking about sneakers.

The score was not miscalibrated. The questions were wrong, and a wrong question produces a confident zero that looks exactly like a real one. Stores whose signals are too thin to rank a product noun still fall back to generic templates, and the run is marked as having used them so the report can say the questions were generic rather than pretending otherwise.

"We asked enough assistants and you were not named" and "we could not ask" are different findings. Only the first is a fact about the brand. The second is a fact about our run, and reporting it as a low score would be a fabrication with a number attached.

So a visibility score is only produced when at least 2 assistants answered at least 4 questions. Below that bar the run reports no score at all. A zero on the index therefore always means the questions were asked, on topic, enough times, and the brand did not come back. A brand we could not measure is omitted from the ranking and counted underneath it, never ranked last.

The same rule governs everything else the engine reports. A check that could not run is not-assessed, which is removed from the denominator rather than scored zero, so a store is never marked down for our network, our timeout, or our expired credential.

A zero means the questions were asked and the brand did not come back. It does not mean the brand never appears. Every measurement has a floor below which a real rate is indistinguishable from absence, and we publish that floor per brand rather than leaving a reader to assume it is zero.

The same question goes to every assistant, so their answers are not independent observations. We measured how much they cluster across 4,329 answered questions and 1,196 question groups: the intra-cluster correlation is 0.594, which discounts a run’s observations by a factor of 2.556. A brand asked four questions of four assistants therefore has about six effective observations, and the smallest rate that could be told apart from zero at 95% confidence is 38%. Each brand page states its own figure, because the number depends on how many questions that catalogue supported.

This is the least flattering number on the site and it is the one that makes the rest of it usable. A ranking that does not say what it could not have seen is asking to be read as more precise than it is.

Tracking a brand over time and measuring it well are different jobs, and they want opposite things. A trend line needs a constant: if the questions change between runs, a movement in the number could be the brand or could be us. A measurement wants to stay current: catalogues change, and a question set frozen forever slowly stops describing what a store sells.

So we do both. Four questions per brand are fixed at its first run and never change; the change column is computed on those alone, so it is stable by construction. Every other question sits in a pool that grows over time. Adding to the pool retires nothing and moves no history, because the core it would have to disturb is untouched and a run stays comparable to an earlier one on the questions both asked.

Roughly a third of measured brands have catalogues too thin to support the full four, and those brands say so on their own page rather than presenting a two-question trend as though it were a four-question one.

These are not edge cases we intend to fix. They are accepted limits of a question set built from a store’s own published taxonomy, and we would rather a reader know them than discover them.

• A small line can outrank the main one. We read what a catalogue declares, not how much of it sells. A skincare brand with a few fragrances can be asked about colognes. Every check we have passes, because cologne really is in the catalogue; what we cannot see is that it is a rounding error of the business.
• Price anchors follow the catalogue, including its bundles. A question can read “under $725” when a product line is full of multi-item kits. The figure is the true median of that line and still reads as absurd.
• An attribute can be real and still be odd. We require an attribute to appear on a product of the type being asked about, which removed questions like “the best fleece bags” from a laptop-bag company. It does not stop a true but unrepresentative pairing, such as asking a key-organiser brand about insulated bags.
• Eleven brands cannot be measured at all. Their published taxonomy names a flavour, a container or a merchandise line rather than a product, or the store is too small to read. We report them as unmeasured rather than scoring them, and for most of them the fact itself is the finding: a catalogue that does not name what it sells is hard for an assistant to enumerate too.

Every measurement carries the conditions it was taken under: which assistants answered, which did not, how many questions were scored. Not as a disclaimer, as part of the result. Two scores taken under different coverage are not the same kind of number, and a reader who cannot see the difference has no way to know that.

This matters most when it is inconvenient. A vendor outage on our side narrows coverage, and the honest response is to say so on the affected runs rather than publish a thinner measurement that looks identical to a full one.

What actually happens when an assistant is unreachable is this. The run is NOT abandoned: it proceeds with the assistants that answered, and the run record is stamped partial rather than ok, carrying how many of the 4 were reached. Anything computed from a partial run is shown as partial on the index, and the condition raises an operational alert (category_pool_degraded) so a narrowed panel is something a person is told about rather than something a reader has to notice.

The difference between refusing and recording matters to a reader. A refusal would mean the index silently stops moving during an outage, which is invisible. Recording the run as partial means the index keeps moving and says, on the affected rows, that it was measured under a narrower panel than usual.

Nobody pays to be here and nobody is here by invitation. There are three ways a store enters, and all three end at the same checks.

• Systematic sourcing. We take domains from named public lists of direct-to-consumer stores. The list a domain came from is recorded against it, so a question about whether the population is skewed by its sources is answerable rather than a matter of trust.
• Submission. Anybody can submit a store on the index page. The form states that submitting lists the store publicly once it passes the checks, and records the answer; a submission without that consent is kept and never published.
• Extension suggestion. The Cresva browser extension has a button offering the store being looked at. It sends the host and the id of the scan, carries the same consent sentence, and never sends a name or an email address, because it never asks for one.

The three checks

A store is admitted when it is REACHABLE, when it SELLS ONLINE, and when we can READ A PRODUCT CATALOGUE from it. That is the whole gate. Each check records what was observed and each failure is a statement about what could be read on the day it was read, which is why a failure is re-run rather than appealed: unreadable stores are asked again every week.

Note what is NOT a condition: how big the store is, how much it sells, whether it is a customer, and how many questions its own catalogue could support. The last one used to be a gate and was wrong, because index scoring is pooled and a brand’s own catalogue yield never enters into it.

Admitted, then scored at the category’s next run

Every admitted brand is pooled and scored against its category’s shared question set at that category’s next run. Since the cron takes one category a night, that is at most 7 nights away and the row says which date. A brand admitted into a category whose catalogue we cannot confidently place is listed WITHOUT a category and is not pooled until an operator assigns one, which is a smaller claim than putting it in the wrong category and ranking it there.

Until a brand has a score it is still listed, with its checks shown, and the row says it has not been scored yet. A public index of stores that silently omits the stores it has admitted and not yet measured is describing its own throughput and calling it a market.

Two populations, and every percentile names its own

Two different sets of stores are counted on this site and they answer different questions. The crawled host population is drawn from the Tranco ranking of the web, which is a real measurement of large sites and a poor comparison for a merchant: measured over one crawl of 100, 97 were large multi-brand commerce. The direct-to-consumer registry is this index: stores of the kind a reader here actually runs.

So no percentile on this site is published without naming which population it is a percentile of, and the size of that population is interpolated into the sentence rather than pinned, because a pinned number becomes false the first time the registry grows and the sentence still reads correctly.

The engine reports what tracking is visible in a page's source. That is a useful thing to know and a disqualifying thing to rank on, because it measures visibility of tracking rather than quality of tracking, and those two run in opposite directions as a team matures.

Moving tags into a server-side container is a step up in sophistication: better data quality, less client weight, more control over what is shared. It also removes those tags from the page source. A brand that does the more advanced thing scores lower. On the first brands measured, the lowest tracking score in the set belonged to a store whose tracking is almost certainly better than the highest.

So the column is reported and never ranked. A metric that falls as competence rises is not a metric, and publishing a league table ordered by it would be publishing the inversion as a finding.

There used to be one. It read 41 for Allbirds and 60 for a 46-product indie store, and that was not a calibration problem to be tuned. It was measuring the wrong thing and being framed as another.

The components share no unit. Schema completeness, pixel hygiene, homepage conversion structure and share of assistant recommendation are four different questions, and a weighted mean asserts both that they are commensurable and that ten points of one trade against ten points of another. They are not and it does not.

Worse, it inverted on sophisticated brands by construction. Maturity moves work out of static HTML: tags into server-side containers, rendering into a SPA, content behind a CDN edge. Every one of those lowers what a crawler observes while raising actual capability. Coverage also varied per run, so averaging over whichever collectors happened to succeed made two runs of the same store non-comparable, never mind two different stores.

And it invited exactly the misreading the reports shipped. "Overall audit score of 41 out of 100" reads as "this business is failing", when what was measured was "this homepage exposes little to a document fetch". A reader can act on "0 of 12 schema types present". Nobody can act on 41. The index ranks one named column for the same reason.

Every class below is a real defect this engine shipped. They are published because a measurement nobody can check is a claim, and because the four instances of the first class were found separately and looked like four unrelated bugs until they were put side by side, at which point they were obviously one mistake made four times.

The last item in each entry is the point of this document. An error that has been fixed will recur; an error with an invariant behind it has to get past the invariant first.

Every index run appends a row that is never updated and never deleted, enforced by the database rather than by convention. It carries each collector score, the coverage conditions, the raw finding counts, and the complete text of every assistant answer.

The full answers are kept rather than excerpts because the matcher has already improved once and will again. An excerpt cut around today's match freezes the record at today's matching quality and makes a future correction impossible. Keeping the whole answer means a past zero can be re-examined against a better method instead of being taken on trust.

The pool’s runs, and the version each was asked under

Every pooled question put to every assistant appends its own row, holding the category, the assistant, the question, the model that answered, the brands it named in the order it named them, and whether it answered, refused or errored. A refusal and an error are kept apart from each other and from an answer naming nobody, because those are three different facts and only one of them is about the market.

Each row also carries the QUESTION-SET VERSION it was asked under, currently 2026-08-31-a, and the field is required rather than optional. The reason is specific: a brand that appears this week and not last week may be a fact about the questions rather than about the brand, and after the fact those two are indistinguishable unless every row says which set produced it. An optional field with a default would let a run claim a version it never used, which is worse than not recording one.

So when the question set changes, the change is in the record rather than in a changelog somebody has to believe: rows from before and after carry different versions, and a comparison across that boundary is visibly a comparison across two sets.

The refresh runs once a day and takes the stalest brands each time. No brand is re-measured inside 6 days, so any one brand is re-measured at least 6 days apart. The floor is a minimum: the true gap depends on how many brands are stale at once. Re-running the same brand daily would render assistant run-to-run variance as brand movement, which is the same class of false signal as every failure mode above: a chart of noise is still a fabricated finding. Sampling different brands each time is not that, and it is what keeps the interval from stretching as the registry grows.

Version 1.2, 2026-09-11. Every change below is a change to what this page SAYS. Where it corrects something, the previous wording is quoted, because a correction nobody can see the before of is indistinguishable from an edit.

• Section 0 is new: the index ranks on the category pool, and the per-brand catalogue-derived method is the customer report. Sections 1 and 2c are now marked as describing the per-brand method only.
• The cadence block now states both: one category a night for the index, and the unchanged per-brand floor. The cycle length is counted from the categories rather than written down.
• Section 3 is CORRECTED. It said “the index refuses to refresh at all when fewer assistants are reachable than the bar requires”. No such refusal exists in the code. A narrowed run is recorded as partial, shown as partial, and raises an alert.
• Failure mode f is new: degraded coverage reported as a successful run, eight runs across nine days, with cost falling as coverage fell.
• Failure mode g is new: 705 scans sent with a literal placeholder in place of the brand name, every one recording a non-mention by construction.
• Section 3b is new: the three intake lanes, the three admission checks, what a listed-but-unscored row shows, and the two benchmark populations.
• Section 7 now describes the pool’s own run records and the question-set version each row carries.

Version 1.1, 2026-08-06: the cadence became derived rather than restated, the panel was reconciled against what actually answered, and a landing-page score whose unit is unknown is withheld rather than printed.

If a measurement here is wrong about your brand, the useful thing to send is the question we asked and what your own check returned. Both are on your report page, which lists every question put to every assistant. A correction that changes the method bumps the version at the top of this document, and a correction that changes what a score means bumps the rubric instead.