AI / ML
Statistical Significance
The threshold at which experimental results are unlikely to have occurred by random chance. Conventionally set at 95% confidence (p-value < 0.05). Running A/B tests or budget changes before reaching statistical significance leads to false conclusions and wasted spend. The required sample size depends on the expected effect size and baseline conversion rate. Small differences in performance require much larger samples to detect reliably. Rushing to conclusions is one of the most expensive mistakes in performance marketing.
Why it matters
It is the guard against acting on noise, and it is routinely read backwards: significance is about the chance of seeing this result if there were no effect, not the chance the variant is better.
In practice
Set the sample size before starting. Checking daily until it turns significant guarantees a significant result eventually, whether or not anything is true.
Related terms