What is the most common AI visibility measurement error?

Attributing an engine failure to the merchant: something on the measuring side fails, and the report describes it as a fact about the store. It is common because the sentence usually names no subject, so the reader supplies one, and the subject a reader supplies is themselves.

Why is a score of zero risky?

Because a zero produced by a broken matcher looks identical to a zero produced by genuine absence. Ten Thousand scored zero on a run where assistants had recommended it, because the matcher was looking for the brand name without its space. A zero should only be reported once the matcher has been shown capable of a hit.

Is scoring a page on the presence of a tag wrong?

It is unverified. The tag existing does not mean the URL inside it resolves or the image renders, and an HTTP 200 does not mean a file exists, because a catch-all rewrite answers 200 for anything. Following the pointer is part of the check.

Does publishing your own errors mean the measurement is unreliable?

It means it is checkable. Every method has failure modes; the difference is whether they are written down with the instance that produced them. These five have named rules against them enforced in named places, which is a smaller claim than correctness and is the one we can support.

Measuring AI Visibility Honestly: Five Ways to Get It Wrong

Every tool that tells you how visible your brand is to AI assistants is doing the same thing underneath: asking some questions, reading some answers, and comparing what came back against what it expected. All three steps fail quietly, and when they do the output is not an error message. It is a confident number.

These are five classes of error we have shipped while building the Brand Index, each with the actual defect that produced it. They are published on our methodology page and repeated here because they generalise: if you are buying a visibility tool, or building one, these are the five questions worth asking of it.

1. Engine failure attributed to the merchant

The shape: something on the measuring side did not work, and the report described that as a fact about the store.

Allbirds was told it had no Product schema, so assistants could not parse its product data. We had scanned the homepage. Homepages do not carry Product schema. It was on the product pages the whole time. Another store was told it refuses automated browsers, when what actually happened was our screenshot vendor being rate limited by that store's CDN on the vendor's shared IP pool; our own crawler got a 200 on every attempt. A third was told the Meta Ad Library returned nothing usable, which reads as a fact about their ad account. Our access token had been dead for 182 days.

The rule that fixes it: a check may emit a claim about a store only from an observation that succeeded, on the page that observation belongs on. When an observation fails or was never attempted, it emits a coverage statement that locates the failure with the measurer. Not-assessed is a third state, distinct from pass and from fail, and it comes out of the denominator rather than scoring zero.

2. Unverified success credited to the merchant

The shape: the same error in the flattering direction, which is why it survives longer. Nobody files a complaint about a score that is too high.

We scored `og:image` on the presence of the tag. The tag is on virtually every modern site, so the points were close to free, and a page whose card renders blank in Slack scored identically to one that renders. We scored `canonical` the same way: the tag exists, so the URL inside it was assumed to resolve. `llms.txt` was worth more than any other single item and was awarded on an HTTP 200, and a catch-all rewrite answers 200 for a file that does not exist. Sitemap presence was inferred from a 200 rather than from parsing what came back.

The rule that fixes it: a check may return ok only when it observed the thing it claims, and not-ok only when it observed the absence. Following the pointer is part of the check, not an optimisation on top of it. Anything else is not-assessed.

Presence of a pointer is not proof of the thing pointed at. Every score awarded on a tag rather than on what the tag leads to is a claim nobody checked.

3. Aggregation error

The shape: the arithmetic itself is wrong, in a way that is visible on the page if anyone looks.

One of our reports stated that 104 of 102 product titles had an issue. A count larger than the population it is drawn from. The same store was told that 61 of 102 products had a missing product image; ground truth from its own products.json was six missing images and about fifty-five reused ones, which is a different problem with a different fix.

The cause was grouping issues by field rather than by field and issue type, then counting issues rather than distinct products. A product with two title problems counted twice. Four different image checks collapsed into one bucket, kept the total, and described it with whichever issue happened to be seen first.

The rule that fixes it: any count presented as N of M counts distinct members of M, and the label names the specific issue type rather than the field it belongs to. A total that can exceed its own population is a bug by construction, not a rounding question.

4. Prose leak

The shape: the rule is enforced where it was written, and not where the text a reader actually sees gets generated.

We validated collector reasons against the no-blaming-the-merchant rule from the first failure above. We did not validate the synthesised report. So a model writing prose from those same collectors could reintroduce exactly the claim the validator existed to catch, in its own words. Vendor detail reached public copy through the same gap: HTTP statuses, JSON error bodies, request ids and billing text are all things a collector strips at source and a generated paragraph can put back.

The rule that fixes it: generated prose is validated against the same invariant as the structured fields it is written from. Vendor detail is replaced wholesale at a single chokepoint rather than patched, because a partial scrub is a leak waiting to be found again.

This one generalises past visibility tooling. Anywhere a model writes the customer-facing sentence from validated data, the validated surface and the published surface are different surfaces, and only one of them is being checked.

5. Matching error

The shape: the measurement is taken correctly and then compared wrongly, which produces a confident zero. This is the worst of the five, because a zero looks like a finding.

Ten Thousand scored 0 on a run where assistants had in fact recommended it. Our brand name was derived from a domain label, and domains have no spaces, so we were looking for `Tenthousand` while every assistant wrote `Ten Thousand`. Separately, catalogue text arrives HTML-encoded, and a literal apostrophe and `'` are different strings, so terms harvested from product data failed to match the same words written normally.

Both are the same mistake: assuming one canonical form on both sides of a comparison whose two sides come from different systems. A domain label against natural prose. Encoded markup against decoded text.

The rule that fixes it: both sides of any comparison get normalised first, and a name match permits the separators a human would insert while still requiring every character in order. A zero is only reported once the matcher has been shown capable of a hit. And the full answer text is stored, so when the matcher improves, history can be re-scored rather than left frozen at the old matching quality.

The five questions to ask a visibility tool

When your check fails, what does my report say? If the answer is a zero rather than a coverage note, failure one is live.
Do you follow the pointers, or score the tags? Ask specifically about `og:image`, `canonical`, `llms.txt` and sitemap.
Can any of your counts exceed their own population? Ask to see an N of M number and what the M is.
Is the paragraph I read validated, or only the data behind it? If a model writes the summary, this is a different surface.
How do you know your matcher can find my brand at all? A tool that has never demonstrated a hit for your name cannot distinguish absence from a matching bug.

None of this means our measurement is correct now. Five documented failures is a record of what we have caught, not evidence that nothing else is wrong, and the fifth one was fixed once before in a different form and came back. What the list supports is narrower: these five have specific rules against them, the rules are enforced in named places, and both the rules and the defects that prompted them are published so somebody else can check.

A measurement nobody can check is a claim. That is as true of ours as of anyone's, which is why the methodology is versioned and the index shows the brands we cannot read alongside the ones we can.

See the method before you see the number The versioned methodology carries all five failure classes with the defects that produced them, plus what each rule is enforced by.