---
title: 'A/B testing'
description: 'A randomized controlled experiment that exposes comparable groups to different variants and compares a predefined outcome.'
canonical_url: 'https://darkfactory.dev/glossary/ab-testing'
markdown_url: 'https://darkfactory.dev/glossary/ab-testing.md'
collection: glossary
date_published: '2026-08-04T00:00:00-04:00'
date_modified: '2026-08-26T00:00:00-04:00'
---

# A/B testing


## Definition

A/B testing is a randomized controlled experiment that assigns comparable units, such as users, sessions, teams, or requests, to different variants and compares a predefined outcome. Random assignment is what lets the experiment estimate the effect of the variant rather than merely observe a correlation.

Design choices determine whether that estimate is credible. Define the population, unit of randomization, primary outcome, guardrail metrics, sample-size plan, and stopping rule before looking at results. Account for spillovers when one participant can affect another, and avoid treating repeated peeks as free decisions.

For AI systems, a winning variant should improve the intended outcome without quietly worsening safety, latency, cost, or a vulnerable subgroup. A statistically detectable effect may still be too small or too costly to matter operationally.

## Distinguish it from nearby terms

An offline benchmark compares systems on a fixed dataset or protocol. A/B testing estimates the effect of variants in an actual user or operational setting. An observational comparison lacks randomized assignment and needs stronger assumptions about why groups differ.

## Check your understanding

A team sends the new assistant to expert users and the old one to everyone else, then calls higher retention an A/B win. The groups were not randomized, so user expertise is a competing explanation.

## Also called

split testing

## Related terms

- [Evaluation (eval)](https://darkfactory.dev/glossary/evaluation)
- [Benchmark](https://darkfactory.dev/glossary/benchmark)

## Evidence and further reading

- [Google Analytics: A/B Testing](https://support.google.com/analytics/answer/13468470?hl=en)