Definition
A/B testing is a randomized controlled experiment that assigns comparable units, such as users, sessions, teams, or requests, to different variants and compares a predefined outcome. Random assignment is what lets the experiment estimate the effect of the variant rather than merely observe a correlation.
Design choices determine whether that estimate is credible. Define the population, unit of randomization, primary outcome, guardrail metrics, sample-size plan, and stopping rule before looking at results. Account for spillovers when one participant can affect another, and avoid treating repeated peeks as free decisions.
For AI systems, a winning variant should improve the intended outcome without quietly worsening safety, latency, cost, or a vulnerable subgroup. A statistically detectable effect may still be too small or too costly to matter operationally.
Distinguish it from nearby terms
An offline benchmark compares systems on a fixed dataset or protocol. A/B testing estimates the effect of variants in an actual user or operational setting. An observational comparison lacks randomized assignment and needs stronger assumptions about why groups differ.
Check your understanding
A team sends the new assistant to expert users and the old one to everyone else, then calls higher retention an A/B win. The groups were not randomized, so user expertise is a competing explanation.