Measure whether the performance gap between two variants (title, snippet, layout, CTA) is statistically significant or just noise.
If the two intervals overlap, the observed gap is probably not significant at the 5% threshold.
A result is called significant at the α=5% threshold if the probability of observing a gap that large by chance (when the two variants are actually equivalent) is less than 5%. That is the p-value. A non-significant test does not say the variants are identical, it says there isn't enough data to conclude. You need either more traffic, or accept that the difference is not actionable.
On-page SEO changes (title, meta description, H1, schema markup) often have invisible effects on ranking itself but measurable effects on CTR. A/B testing lets you quantify the effect objectively. Limit: SEO is not a true A/B environment, you cannot serve two different titles to the same user. Practical method, temporal split test (variant A for 4 weeks, variant B for the next 4) with seasonal adjustment, or split by cluster of comparable URLs.
Depends on the baseline and the effect you want to detect. Rule of thumb: to detect a 10% lift on a 5% conversion rate, you need about 6,000 visitors per variant (12k total) at 95% confidence, 80% power. The lower the base rate and the smaller the expected lift, the more traffic you need. For an SEO test on a page at 100 visits/day, count on 2 months minimum for a usable signal.
This tool uses a Z-test for two proportions (normal approximation, ideal when n*p > 5 and n*(1-p) > 5, in practice beyond a few hundred visitors per variant). Chi² is mathematically equivalent in 2x2. Bayesian methods (probability that B > A) are more intuitive but less standard for business reporting, we will add them in a V2 if there is demand.
Our network catalog is browsable without signup. Publisher pricing shown, no commission, no middleman.