Why it matters to a buyer
Multivariate testing sounds like a more thorough version of an A/B test, but for most buyers it is the wrong tool: because it splits traffic across every combination of the elements being tested, it needs far more total visitors than a simple two-version test to reach a reliable conclusion on any one combination, which puts it out of reach for most small and mid-sized sites. Understanding this distinction stops a buyer from being sold a multivariate program the site's traffic cannot actually support.
How it is measured or applied
A multivariate test is set up by choosing a small number of elements to vary, each with a small number of variations, since the total combinations multiply quickly and each needs enough traffic on its own. It is applied on genuinely high-traffic pages, where the combined sample can be split many ways and still reach significance for each combination in a reasonable time, and read using the testing platform's own statistical analysis, which reports both each element's individual effect and any interaction between elements. On lower traffic, a sequence of simple A/B tests, one variable at a time, reaches a usable answer faster.
Sources
- Multivariate testing, Optimizely
