Why it matters to a buyer
An SRM is a silent way an otherwise well-designed test can produce a misleading result: it usually points to a bug in how visitors are assigned or tracked, such as a bot skewing one variant, a caching layer serving the wrong version inconsistently, or a redirect that fails more often for one variant than the other, and any of those can bias the outcome in ways that have nothing to do with which version is genuinely better. A buyer should ask whether an agency checks for SRM on every test result, not just whether a result reached significance.
How it is measured or applied
SRM is detected with a simple statistical check, typically a chi-square test, comparing the actual visitor counts in each variant against the intended split, flagged when the deviation is far too large to be explained by normal random chance. Microsoft's own experimentation platform runs this check automatically before any test result is trusted, because a mature testing culture treats it as a mandatory gate rather than an optional check. When SRM is found, the test's result is discarded rather than reported, and the underlying assignment or tracking bug is found and fixed before the test is rerun.
Sources
- Diagnosing Sample Ratio Mismatch in A/B Testing, Microsoft Research
