A/B Testing for Data Sciences
A/B testing is a randomised experiment that compares two versions of a product change to measure its effect on a metric. This unit covers hypotheses, choosing metrics, calculating sample size and power, randomisation, running the test, and analysing results without common errors such as peeking or multiple comparisons.
Stage: Experimentation and Causal Thinking · Part of the Data Science roadmap · Last reviewed 19 November 2026
What to learn in A/B Testing
- Write a testable hypothesis with a primary metric and guardrail metrics
- Calculate required sample size from baseline rate, minimum detectable effect and power
- Check randomisation with a sample ratio mismatch test
- Analyse a conversion test with a two-proportion z-test or confidence interval
- Explain why repeatedly peeking at results inflates false positives
A/B Testing interview questions
What is statistical power in an A/B test?
Power is the probability that the test detects a real effect of the planned size. Low power means true improvements are often missed. It depends on sample size, effect size, variance and the significance level, so you set it before launching.
What is a sample ratio mismatch and why does it matter?
It is when the observed split of users between variants differs from the planned split by more than chance, for example 50/50 planned but 52/48 seen at large scale. It signals a bug in assignment or logging, so results should not be trusted until it is explained.
Where A/B Testing fits in the Data Science roadmap
A/B Testing belongs to the Experimentation and Causal Thinking stage (intermediate level), which takes about 4 hours. What actually works.
Other units in this stage
Back to the full Data Science roadmap