Rooted in Francis Bacon (Novum Organum, 1620) and John Stuart Mill (A System of Logic, 1843), made rigorous by Ronald A. Fisher (Statistical Methods for Research Workers, 1925), and set out for online tests by Ron Kohavi, Diane Tang, and Ya Xu (Trustworthy Online Controlled Experiments, 2020).
Assigns units at random to a change or a control, measures an outcome fixed in advance, and estimates whether the change caused the difference.
When to reach for it
When a decision needs proof that a change causes a result, the change can be randomized, and the result can be measured cleanly.
How it runs
- State the hypothesis: this change causes that outcome.
- Fix the metric, test, sample size, and stopping rule in advance.
- Run a power calculation; rethink if the sample is infeasible.
- Assign units at random and run without peeking.
- Analyze as planned, and report the effect size with intervals.
- Replicate before scaling up.
Where it sits in SOLVE eX
Go to the source
- Novum Organum, 1620
- A System of Logic, 1843
- Statistical Methods for Research Workers, Oliver and Boyd, 1925
- Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press, 2020