Why it matters to a buyer
An A/B test is the only reliable way to know whether a change to a page helped, because it removes seasonality, campaigns and chance from the comparison. Without one, a redesign that coincides with a busy month looks like a win. For a buyer, the catch is traffic: a test on a page with few conversions can take months to reach a trustworthy result, so an honest agency will say which pages have enough volume to test and which should simply be fixed.
How it is measured or applied
A test starts with a written hypothesis, a single primary metric and a sample size calculated in advance from the current conversion rate and the smallest lift worth detecting. The test runs for whole weeks until that sample is reached, and the result is read once, not checked daily and stopped when it looks good. Google's guidance for search is to use rel canonical on variant URLs, use 302 redirects rather than 301 where redirects are involved, avoid cloaking, and end the test promptly once it is decided.
Sources
- Minimize A/B testing impact in Google Search, Google Search Central
