We here at Win-Vector LLC been working through an ad-hoc series about A/B testing combining elements of both operations research and statistical points of view.

• A dynamic programming solution to A/B test design
• Why does designing a simple A/B test seem so complicated?
• A clear picture of power and significance in A/B tests
• Bandit Formulations for A/B Tests: Some Intuition
• Bayesian/loss-oriented: New video course: Campaign Response Testing

Our most recent article was a dynamic programming solution to the A/B test problem. Explicitly solving such dynamic programs gets long and tedious, so you are well served by finding and introducing clever invariants to track (something better than just raw win-rates). That clever idea is called “sequential analysis” and was introduced by Abraham Wald (somebody we have written about before). If you have ever heard of a test plan such as “first process to get more than 30 wins ahead of the other is the one we choose” you have seen methods derived from Wald’s sequential analysis technique.

Wald’s famous airplane armor problem

A corrected version of the detailed article is now here.

Categories: Mathematics

Tagged as: A/B testing math programming R Statistics Statistics as it should be Wald

jmount

Data Scientist and trainer at Win Vector LLC. One of the authors of Practical Data Science with R.